[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:54:16 dist-test-slave-x1lh krb5kdc[20946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:54:16 dist-test-slave-x1lh krb5kdc[20946](info): set up 2 sockets
Sep 25 22:54:16 dist-test-slave-x1lh krb5kdc[20946](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 22:54:17.145344 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:54:17 dist-test-slave-x1lh krb5kdc[20946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:54:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:54:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:54:17.273188 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:44829
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:37885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:44829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:17.385488 20959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:17.385798 20959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:17.385875 20959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:17.389899 20959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:17.389997 20959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:17.390036 20959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:17.390074 20959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:17.390110 20959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:17.395470 20959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:37885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:44829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:44829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:17.396662 20959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:17.397802 20959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:17.406237 20964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:17.406294 20967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:17.406564 20959 server_base.cc:1034] running on GCE node
W20260925 22:54:17.408077 20965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:17.408744 20959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:17.409989 20959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:17.411207 20959 hybrid_clock.cc:648] HybridClock initialized: now 1790376857411183 us; error 41 us; skew 500 ppm
Sep 25 22:54:17 dist-test-slave-x1lh krb5kdc[20946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376857, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:17.414304 20959 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:54:17.415307 20959 webserver.cc:460] Webserver started at http://127.20.114.254:43375/ using document root <none> and password file <none>
I20260925 22:54:17.415858 20959 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:17.415918 20959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:17.416090 20959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:17.418303 20959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "4230aab993c049168240189a932ee847"
format_stamp: "Formatted at 2026-09-25 22:54:17 on dist-test-slave-x1lh"
server_key: "f7b2256b3ecccf7a442297ca83af5f81"
server_key_iv: "17436840c722957fd0a7bc3045528a0a"
server_key_version: "encryptionkey@0"
I20260925 22:54:17.418859 20959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "4230aab993c049168240189a932ee847"
format_stamp: "Formatted at 2026-09-25 22:54:17 on dist-test-slave-x1lh"
server_key: "f7b2256b3ecccf7a442297ca83af5f81"
server_key_iv: "17436840c722957fd0a7bc3045528a0a"
server_key_version: "encryptionkey@0"
I20260925 22:54:17.422880 20959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:54:17.425328 20974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:17.426656 20959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:17.426784 20959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "4230aab993c049168240189a932ee847"
format_stamp: "Formatted at 2026-09-25 22:54:17 on dist-test-slave-x1lh"
server_key: "f7b2256b3ecccf7a442297ca83af5f81"
server_key_iv: "17436840c722957fd0a7bc3045528a0a"
server_key_version: "encryptionkey@0"
I20260925 22:54:17.426893 20959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:17.467877 20959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:17.471050 20959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:17.471266 20959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:17.478410 20959 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:44829
I20260925 22:54:17.478431 21026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:44829 every 8 connection(s)
I20260925 22:54:17.479419 20959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:54:17.482196 21027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:17.487850 21027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4230aab993c049168240189a932ee847: Bootstrap starting.
I20260925 22:54:17.489943 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 20959
I20260925 22:54:17.490108 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:54:17.490823 21027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4230aab993c049168240189a932ee847: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:17.491742 20939 external_mini_cluster.cc:1428] Setting key dd980f4114e6e5506e08bde0a98575ab
I20260925 22:54:17.491704 21027 log.cc:824] T 00000000000000000000000000000000 P 4230aab993c049168240189a932ee847: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:17.494159 21027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4230aab993c049168240189a932ee847: No bootstrap required, opened a new log
I20260925 22:54:17.499182 21027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4230aab993c049168240189a932ee847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4230aab993c049168240189a932ee847" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44829 } }
I20260925 22:54:17.499473 21027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4230aab993c049168240189a932ee847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:17.499526 21027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4230aab993c049168240189a932ee847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4230aab993c049168240189a932ee847, State: Initialized, Role: FOLLOWER
I20260925 22:54:17.500005 21027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4230aab993c049168240189a932ee847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4230aab993c049168240189a932ee847" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44829 } }
I20260925 22:54:17.500164 21027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4230aab993c049168240189a932ee847 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:54:17.500234 21027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4230aab993c049168240189a932ee847 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:54:17.500341 21027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4230aab993c049168240189a932ee847 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:17.501410 21027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4230aab993c049168240189a932ee847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4230aab993c049168240189a932ee847" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44829 } }
I20260925 22:54:17.501770 21027 leader_election.cc:302] T 00000000000000000000000000000000 P 4230aab993c049168240189a932ee847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4230aab993c049168240189a932ee847; no voters:
I20260925 22:54:17.502111 21027 leader_election.cc:288] T 00000000000000000000000000000000 P 4230aab993c049168240189a932ee847 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:54:17.502193 21032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4230aab993c049168240189a932ee847 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:17.502425 21032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4230aab993c049168240189a932ee847 [term 1 LEADER]: Becoming Leader. State: Replica: 4230aab993c049168240189a932ee847, State: Running, Role: LEADER
I20260925 22:54:17.502735 21032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4230aab993c049168240189a932ee847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4230aab993c049168240189a932ee847" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44829 } }
I20260925 22:54:17.503443 21027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4230aab993c049168240189a932ee847 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:17.503803 21032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4230aab993c049168240189a932ee847 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4230aab993c049168240189a932ee847. Latest consensus state: current_term: 1 leader_uuid: "4230aab993c049168240189a932ee847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4230aab993c049168240189a932ee847" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44829 } } }
I20260925 22:54:17.503933 21032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4230aab993c049168240189a932ee847 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:17.504333 21036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:54:17.504345 21033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4230aab993c049168240189a932ee847 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4230aab993c049168240189a932ee847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4230aab993c049168240189a932ee847" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44829 } } }
I20260925 22:54:17.504536 21033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4230aab993c049168240189a932ee847 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:17.507467 21036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 22:54:17 dist-test-slave-x1lh krb5kdc[20946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:17.513242 21036 catalog_manager.cc:1347] Generated new cluster ID: fc723d0011f54736af4d45999e95c483
I20260925 22:54:17.513302 21036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:17.519531 21030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:17.495216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34778 (local address 127.20.114.254:44829)
0925 22:54:17.495655 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:17.495675 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:17.496971 (+ 1296us) server_negotiation.cc:407] Connection header received
0925 22:54:17.500676 (+ 3705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:17.500689 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:17.501050 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:17.501318 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:17.504206 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:17.505871 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:17.507589 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:17.508238 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:17.508362 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:17.513731 (+ 5369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:17.513738 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:17.513756 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:17.513792 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:17.516034 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:17.516636 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:17.516642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:17.516652 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:17.516757 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:17.517143 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:17.517147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:17.517149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:17.517554 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:17.517779 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:17.518338 (+ 559us) server_negotiation.cc:299] Negotiation successful
0925 22:54:17.518614 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":119,"threads_started":1}
I20260925 22:54:17.535029 21036 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:54:17.536202 21036 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:17.543919 21036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4230aab993c049168240189a932ee847: Generated new TSK 0
I20260925 22:54:17.544708 21036 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.114.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
I20260925 22:54:17.580147 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44829
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:37885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:17.690953 21053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:17.691210 21053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:17.691255 21053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:17.695328 21053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:17.695377 21053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:17.695473 21053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:54:17.700570 21053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:37885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44829
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:17.701591 21053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:17.702615 21053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:17.712507 21061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:17.712507 21058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:17.712523 21059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:17.712739 21053 server_base.cc:1034] running on GCE node
I20260925 22:54:17.713259 21053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:17.713798 21053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:17.714957 21053 hybrid_clock.cc:648] HybridClock initialized: now 1790376857714943 us; error 27 us; skew 500 ppm
Sep 25 22:54:17 dist-test-slave-x1lh krb5kdc[20946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376857, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:17.717705 21053 init.cc:377] Logged in from keytab as kudu/127.20.114.193@KRBTEST.COM (short username kudu)
I20260925 22:54:17.718614 21053 webserver.cc:460] Webserver started at http://127.20.114.193:43813/ using document root <none> and password file <none>
I20260925 22:54:17.719158 21053 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:17.719251 21053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:17.719491 21053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:17.721504 21053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "be09ca3588a64589948076511b7a39ec"
format_stamp: "Formatted at 2026-09-25 22:54:17 on dist-test-slave-x1lh"
server_key: "a4a3e347ff6589d1100194db88d948bc"
server_key_iv: "5761a6fa14d3f3e6244d1e302198c203"
server_key_version: "encryptionkey@0"
I20260925 22:54:17.722069 21053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "be09ca3588a64589948076511b7a39ec"
format_stamp: "Formatted at 2026-09-25 22:54:17 on dist-test-slave-x1lh"
server_key: "a4a3e347ff6589d1100194db88d948bc"
server_key_iv: "5761a6fa14d3f3e6244d1e302198c203"
server_key_version: "encryptionkey@0"
I20260925 22:54:17.725648 21053 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:54:17.728075 21068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:17.729305 21053 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:54:17.729435 21053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "be09ca3588a64589948076511b7a39ec"
format_stamp: "Formatted at 2026-09-25 22:54:17 on dist-test-slave-x1lh"
server_key: "a4a3e347ff6589d1100194db88d948bc"
server_key_iv: "5761a6fa14d3f3e6244d1e302198c203"
server_key_version: "encryptionkey@0"
I20260925 22:54:17.729552 21053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:17.754058 21053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:17.757063 21053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:17.757256 21053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:17.757859 21053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:17.758723 21053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:17.758795 21053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:17.758867 21053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:17.758903 21053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:17.767828 21053 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:33545
I20260925 22:54:17.767860 21181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:33545 every 8 connection(s)
I20260925 22:54:17.768723 21053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:54:17.776605 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 21053
I20260925 22:54:17.776717 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:54:17.777021 20939 external_mini_cluster.cc:1428] Setting key 8e89c96dd54fa3fb3a2bbef1a2f36296
Sep 25 22:54:17 dist-test-slave-x1lh krb5kdc[20946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376857, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:17.782727 21030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:17.770525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:51847 (local address 127.20.114.254:44829)
0925 22:54:17.770670 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:17.770674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:17.771496 (+ 822us) server_negotiation.cc:407] Connection header received
0925 22:54:17.772782 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:17.772785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:17.772844 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:17.772973 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:17.774096 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:17.774624 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:17.775743 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:17.775969 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:17.776083 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:17.778857 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:17.778860 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:17.778862 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:17.778883 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:17.780527 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:17.781090 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:17.781095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:17.781098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:17.781201 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:17.781525 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:17.781531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:17.781536 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:17.781778 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:17.781880 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:17.782352 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 22:54:17.782507 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:54:17.783449 21184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:17.770918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:44829 (local address 127.20.114.193:51847)
0925 22:54:17.771334 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:17.771376 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:17.772574 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:17.773153 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:17.773160 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:17.773497 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:17.773943 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:17.773956 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:17.774762 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:17.774765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:17.775619 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:17.775628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:17.776117 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:17.776121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:17.776281 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:17.776972 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:17.776989 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:17.778685 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:17.780645 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:17.780650 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:17.780662 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:17.780995 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:17.781293 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:17.781295 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:17.781296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:17.781429 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:17.781887 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:17.781892 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:17.782113 (+ 221us) client_negotiation.cc:769] Sending connection context
0925 22:54:17.782267 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 22:54:17.782520 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":102,"threads_started":1}
I20260925 22:54:17.784649 21182 heartbeater.cc:347] Connected to a master server at 127.20.114.254:44829
I20260925 22:54:17.784931 21182 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:17.785425 21182 heartbeater.cc:511] Master 127.20.114.254:44829 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.114.194@KRBTEST.COM; defaulting to no policy
I20260925 22:54:17.787268 20991 ts_manager.cc:194] Registered new tserver with Master: be09ca3588a64589948076511b7a39ec (127.20.114.193:33545)
I20260925 22:54:17.788502 20991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:51847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
I20260925 22:54:17.808565 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44829
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:37885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:17.919986 21187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:17.920255 21187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:17.920331 21187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:17.924284 21187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:17.924378 21187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:17.924461 21187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:54:17.929641 21187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:37885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44829
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:17.930687 21187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:17.931696 21187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:17.941466 21193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:17.941515 21195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:17.941566 21187 server_base.cc:1034] running on GCE node
W20260925 22:54:17.941452 21192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:17.942160 21187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:17.942813 21187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:17.944003 21187 hybrid_clock.cc:648] HybridClock initialized: now 1790376857943987 us; error 27 us; skew 500 ppm
Sep 25 22:54:17 dist-test-slave-x1lh krb5kdc[20946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376857, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:17.946825 21187 init.cc:377] Logged in from keytab as kudu/127.20.114.194@KRBTEST.COM (short username kudu)
I20260925 22:54:17.947745 21187 webserver.cc:460] Webserver started at http://127.20.114.194:41693/ using document root <none> and password file <none>
I20260925 22:54:17.948361 21187 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:17.948432 21187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:17.948645 21187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:17.950709 21187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "6966d23803a645e29aea8e9746de1cde"
format_stamp: "Formatted at 2026-09-25 22:54:17 on dist-test-slave-x1lh"
server_key: "4ae3e0b65bbf286c70ebb0aa1bc1e1e4"
server_key_iv: "c5085795ee72c578b407609d634071b5"
server_key_version: "encryptionkey@0"
I20260925 22:54:17.951268 21187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "6966d23803a645e29aea8e9746de1cde"
format_stamp: "Formatted at 2026-09-25 22:54:17 on dist-test-slave-x1lh"
server_key: "4ae3e0b65bbf286c70ebb0aa1bc1e1e4"
server_key_iv: "c5085795ee72c578b407609d634071b5"
server_key_version: "encryptionkey@0"
I20260925 22:54:17.954975 21187 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:54:17.957269 21202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:17.958513 21187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:17.958639 21187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "6966d23803a645e29aea8e9746de1cde"
format_stamp: "Formatted at 2026-09-25 22:54:17 on dist-test-slave-x1lh"
server_key: "4ae3e0b65bbf286c70ebb0aa1bc1e1e4"
server_key_iv: "c5085795ee72c578b407609d634071b5"
server_key_version: "encryptionkey@0"
I20260925 22:54:17.958739 21187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:17.985059 21187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:17.988534 21187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:17.988752 21187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:17.989413 21187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:17.990242 21187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:17.990288 21187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:17.990355 21187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:17.990394 21187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:17.999987 21187 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:44679
I20260925 22:54:18.000021 21315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:44679 every 8 connection(s)
I20260925 22:54:18.000983 21187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:54:18.004791 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 21187
I20260925 22:54:18.004951 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:54:18.005205 20939 external_mini_cluster.cc:1428] Setting key 60c9ca9c719502465ac19a8031ebcbce
WARNING: no policy specified for kudu/127.20.114.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.195@KRBTEST.COM" created.
Sep 25 22:54:18 dist-test-slave-x1lh krb5kdc[20946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376857, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:18.021941 21030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:18.002934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:57211 (local address 127.20.114.254:44829)
0925 22:54:18.003077 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:18.003080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:18.004044 (+ 964us) server_negotiation.cc:407] Connection header received
0925 22:54:18.005644 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:18.005648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:18.005710 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:18.005820 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:18.007141 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:18.007659 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:18.009476 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:18.009695 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:18.009811 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:18.018131 (+ 8320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:18.018135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:18.018137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:18.018163 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:18.019756 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:18.020288 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:18.020292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:18.020294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:18.020375 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:18.020728 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:18.020731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:18.020733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:18.021002 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:18.021105 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:18.021627 (+ 522us) server_negotiation.cc:299] Negotiation successful
0925 22:54:18.021759 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:54:18.022771 21318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:18.003418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:44829 (local address 127.20.114.194:57211)
0925 22:54:18.003896 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:18.003937 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:18.005415 (+ 1478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:18.006001 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:18.006008 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:18.006497 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:18.006997 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:18.007011 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:18.007813 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:18.007816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:18.009347 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:18.009365 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:18.009846 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:18.009849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:18.010036 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:18.015681 (+ 5645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:18.015701 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:18.017912 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:18.019880 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:18.019886 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:18.019905 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:18.020171 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:18.020478 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:18.020481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:18.020483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:18.020622 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:18.021114 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:18.021123 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:18.021371 (+ 248us) client_negotiation.cc:769] Sending connection context
0925 22:54:18.021575 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 22:54:18.021913 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":148,"threads_started":1}
I20260925 22:54:18.024168 21316 heartbeater.cc:347] Connected to a master server at 127.20.114.254:44829
I20260925 22:54:18.024442 21316 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:18.025038 21316 heartbeater.cc:511] Master 127.20.114.254:44829 requested a full tablet report, sending...
I20260925 22:54:18.026177 20991 ts_manager.cc:194] Registered new tserver with Master: 6966d23803a645e29aea8e9746de1cde (127.20.114.194:44679)
I20260925 22:54:18.026856 20991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:57211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
I20260925 22:54:18.035648 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44829
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:37885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:18.151283 21321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:18.151542 21321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:18.151587 21321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:18.155596 21321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:18.155654 21321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:18.155754 21321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:54:18.161077 21321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:37885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44829
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:18.162168 21321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:18.163273 21321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:18.172636 21329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:18.172628 21327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:18.172638 21326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:18.172921 21321 server_base.cc:1034] running on GCE node
I20260925 22:54:18.173415 21321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:18.174111 21321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:18.175272 21321 hybrid_clock.cc:648] HybridClock initialized: now 1790376858175247 us; error 36 us; skew 500 ppm
Sep 25 22:54:18 dist-test-slave-x1lh krb5kdc[20946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376858, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:18.178076 21321 init.cc:377] Logged in from keytab as kudu/127.20.114.195@KRBTEST.COM (short username kudu)
I20260925 22:54:18.179030 21321 webserver.cc:460] Webserver started at http://127.20.114.195:33683/ using document root <none> and password file <none>
I20260925 22:54:18.179615 21321 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:18.179708 21321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:18.179965 21321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:18.182126 21321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "ab34ffec45364c25a2b43c167551bc33"
format_stamp: "Formatted at 2026-09-25 22:54:18 on dist-test-slave-x1lh"
server_key: "075f7cabda2c4364d9690a65edbd38ce"
server_key_iv: "f5efdc5f09363df8e702fa76c33fcc8d"
server_key_version: "encryptionkey@0"
I20260925 22:54:18.182691 21321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "ab34ffec45364c25a2b43c167551bc33"
format_stamp: "Formatted at 2026-09-25 22:54:18 on dist-test-slave-x1lh"
server_key: "075f7cabda2c4364d9690a65edbd38ce"
server_key_iv: "f5efdc5f09363df8e702fa76c33fcc8d"
server_key_version: "encryptionkey@0"
I20260925 22:54:18.186510 21321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:54:18.188956 21336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:18.190114 21321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:18.190248 21321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "ab34ffec45364c25a2b43c167551bc33"
format_stamp: "Formatted at 2026-09-25 22:54:18 on dist-test-slave-x1lh"
server_key: "075f7cabda2c4364d9690a65edbd38ce"
server_key_iv: "f5efdc5f09363df8e702fa76c33fcc8d"
server_key_version: "encryptionkey@0"
I20260925 22:54:18.190359 21321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:18.205729 21321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:18.209098 21321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:18.209306 21321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:18.209890 21321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:18.210700 21321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:18.210778 21321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:18.210866 21321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:18.210920 21321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:18.220000 21321 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:33651
I20260925 22:54:18.220021 21449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:33651 every 8 connection(s)
I20260925 22:54:18.220907 21321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:54:18.222021 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 21321
I20260925 22:54:18.222143 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:54:18.222440 20939 external_mini_cluster.cc:1428] Setting key 2d755681f006694ef343204fc79712e4
Sep 25 22:54:18 dist-test-slave-x1lh krb5kdc[20946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376858, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:18.234277 21030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:18.222914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:58941 (local address 127.20.114.254:44829)
0925 22:54:18.223037 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:18.223046 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:18.223994 (+ 948us) server_negotiation.cc:407] Connection header received
0925 22:54:18.225420 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:18.225423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:18.225467 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:18.225529 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:18.226611 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:18.227145 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:18.228305 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:18.228513 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:18.228584 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:18.231077 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:18.231080 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:18.231082 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:18.231103 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:18.232362 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:18.232848 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:18.232851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:18.232853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:18.232991 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:18.233283 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:18.233285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:18.233286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:18.233468 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:18.233541 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:18.234051 (+ 510us) server_negotiation.cc:299] Negotiation successful
0925 22:54:18.234159 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 22:54:18.235095 21452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:18.223367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:44829 (local address 127.20.114.195:58941)
0925 22:54:18.223844 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:18.223909 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:18.225230 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:18.225667 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:18.225673 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:18.226022 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:18.226496 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:18.226509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:18.227247 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:18.227250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:18.228205 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:18.228214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:18.228630 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:18.228633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:18.228766 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:18.229464 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:18.229480 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:18.230922 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:18.232491 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:18.232499 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:18.232513 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:18.232762 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:18.233072 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:18.233074 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:18.233075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:18.233203 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:18.233575 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:18.233582 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:18.233814 (+ 232us) client_negotiation.cc:769] Sending connection context
0925 22:54:18.233971 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 22:54:18.234273 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":120,"threads_started":1}
I20260925 22:54:18.236250 21450 heartbeater.cc:347] Connected to a master server at 127.20.114.254:44829
I20260925 22:54:18.236497 21450 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:18.237056 21450 heartbeater.cc:511] Master 127.20.114.254:44829 requested a full tablet report, sending...
I20260925 22:54:18.238040 20991 ts_manager.cc:194] Registered new tserver with Master: ab34ffec45364c25a2b43c167551bc33 (127.20.114.195:33651)
I20260925 22:54:18.238605 20991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:58941
I20260925 22:54:18.248656 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:54:18 dist-test-slave-x1lh krb5kdc[20946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376858, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:54:18 dist-test-slave-x1lh krb5kdc[20946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376858, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.114.193@KRBTEST.COM
I20260925 22:54:18.274646 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:18.265346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53554 (local address 127.20.114.193:33545)
0925 22:54:18.265672 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:18.265686 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:18.265712 (+ 26us) server_negotiation.cc:407] Connection header received
0925 22:54:18.265805 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:18.265812 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:18.266015 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:18.266140 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:18.266766 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:18.267540 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:18.268340 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:18.268709 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:18.268792 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:18.271337 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:18.271346 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:18.271360 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:18.271385 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:18.272855 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:18.273316 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:18.273322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:18.273323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:18.273422 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:18.273684 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:18.273686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:18.273687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:18.273938 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:18.274108 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:18.274251 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 22:54:18.274428 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":71,"threads_started":1}
W20260925 22:54:18.275638 21096 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53554
Sep 25 22:54:18 dist-test-slave-x1lh krb5kdc[20946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:54:18 dist-test-slave-x1lh krb5kdc[20946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.193@KRBTEST.COM
I20260925 22:54:18.301558 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:18.293248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53558 (local address 127.20.114.193:33545)
0925 22:54:18.293391 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:18.293395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:18.293551 (+ 156us) server_negotiation.cc:407] Connection header received
0925 22:54:18.293663 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:18.293666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:18.293756 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:18.293859 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:18.294536 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:18.295042 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:18.295913 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:18.296133 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:18.296206 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:18.298596 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:18.298600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:18.298602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:18.298625 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:18.300040 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:18.300438 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:18.300441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:18.300442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:18.300513 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:18.300786 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:18.300788 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:18.300789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:18.301028 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:18.301128 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:18.301334 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 22:54:18.301435 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:54:18.302613 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 21053
I20260925 22:54:18.308295 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 21187
I20260925 22:54:18.313727 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 21321
I20260925 22:54:18.318768 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 20959
2026-09-25T22:54:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2220 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:54:18 dist-test-slave-x1lh krb5kdc[21464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:54:18 dist-test-slave-x1lh krb5kdc[21464](info): set up 2 sockets
Sep 25 22:54:18 dist-test-slave-x1lh krb5kdc[21464](info): commencing operation
krb5kdc: starting...
W20260925 22:54:19.352667 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:54:19 dist-test-slave-x1lh krb5kdc[21464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:54:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:54:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:54:19.475932 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:38103
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:34597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:38103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:19.585253 21477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:19.585543 21477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:19.585616 21477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:19.591568 21477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:19.591648 21477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:19.591684 21477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:19.591722 21477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:19.591756 21477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:19.597079 21477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:34597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:38103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:38103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:19.598165 21477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:19.599206 21477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:19.607600 21483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:19.607779 21477 server_base.cc:1034] running on GCE node
W20260925 22:54:19.607613 21485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:19.607596 21482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:19.608472 21477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:19.609608 21477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:19.610775 21477 hybrid_clock.cc:648] HybridClock initialized: now 1790376859610758 us; error 30 us; skew 500 ppm
Sep 25 22:54:19 dist-test-slave-x1lh krb5kdc[21464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376859, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:19.613588 21477 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:54:19.614462 21477 webserver.cc:460] Webserver started at http://127.20.114.254:33145/ using document root <none> and password file <none>
I20260925 22:54:19.614984 21477 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:19.615039 21477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:19.615204 21477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:19.617206 21477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "7d68bf93b9e64696b13e23a8b6bcb797"
format_stamp: "Formatted at 2026-09-25 22:54:19 on dist-test-slave-x1lh"
server_key: "cfde95e0dc4a23509b9801d359c0a308"
server_key_iv: "a217fc942ceec94103c16231060b65cd"
server_key_version: "encryptionkey@0"
I20260925 22:54:19.617707 21477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "7d68bf93b9e64696b13e23a8b6bcb797"
format_stamp: "Formatted at 2026-09-25 22:54:19 on dist-test-slave-x1lh"
server_key: "cfde95e0dc4a23509b9801d359c0a308"
server_key_iv: "a217fc942ceec94103c16231060b65cd"
server_key_version: "encryptionkey@0"
I20260925 22:54:19.621276 21477 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:54:19.623485 21492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:19.624679 21477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:19.624818 21477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "7d68bf93b9e64696b13e23a8b6bcb797"
format_stamp: "Formatted at 2026-09-25 22:54:19 on dist-test-slave-x1lh"
server_key: "cfde95e0dc4a23509b9801d359c0a308"
server_key_iv: "a217fc942ceec94103c16231060b65cd"
server_key_version: "encryptionkey@0"
I20260925 22:54:19.624990 21477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:19.642030 21477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:19.645285 21477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:19.645491 21477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:19.652516 21477 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:38103
I20260925 22:54:19.652539 21544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:38103 every 8 connection(s)
I20260925 22:54:19.653446 21477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:54:19.656073 21545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:19.661271 21545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d68bf93b9e64696b13e23a8b6bcb797: Bootstrap starting.
I20260925 22:54:19.661685 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 21477
I20260925 22:54:19.661792 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:54:19.662040 20939 external_mini_cluster.cc:1428] Setting key e5f4bfcaf660097ab1b22bf973ea8922
I20260925 22:54:19.663916 21545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d68bf93b9e64696b13e23a8b6bcb797: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:19.664606 21545 log.cc:824] T 00000000000000000000000000000000 P 7d68bf93b9e64696b13e23a8b6bcb797: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:19.666416 21545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d68bf93b9e64696b13e23a8b6bcb797: No bootstrap required, opened a new log
I20260925 22:54:19.669852 21545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d68bf93b9e64696b13e23a8b6bcb797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d68bf93b9e64696b13e23a8b6bcb797" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38103 } }
I20260925 22:54:19.670101 21545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d68bf93b9e64696b13e23a8b6bcb797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:19.670171 21545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d68bf93b9e64696b13e23a8b6bcb797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d68bf93b9e64696b13e23a8b6bcb797, State: Initialized, Role: FOLLOWER
Sep 25 22:54:19 dist-test-slave-x1lh krb5kdc[21464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:19.670647 21545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d68bf93b9e64696b13e23a8b6bcb797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d68bf93b9e64696b13e23a8b6bcb797" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38103 } }
I20260925 22:54:19.670809 21545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d68bf93b9e64696b13e23a8b6bcb797 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:54:19.670878 21545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d68bf93b9e64696b13e23a8b6bcb797 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:54:19.670998 21545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d68bf93b9e64696b13e23a8b6bcb797 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:19.672004 21545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d68bf93b9e64696b13e23a8b6bcb797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d68bf93b9e64696b13e23a8b6bcb797" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38103 } }
I20260925 22:54:19.672360 21545 leader_election.cc:302] T 00000000000000000000000000000000 P 7d68bf93b9e64696b13e23a8b6bcb797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d68bf93b9e64696b13e23a8b6bcb797; no voters:
I20260925 22:54:19.672678 21545 leader_election.cc:288] T 00000000000000000000000000000000 P 7d68bf93b9e64696b13e23a8b6bcb797 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:54:19.672786 21550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d68bf93b9e64696b13e23a8b6bcb797 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:19.673035 21550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d68bf93b9e64696b13e23a8b6bcb797 [term 1 LEADER]: Becoming Leader. State: Replica: 7d68bf93b9e64696b13e23a8b6bcb797, State: Running, Role: LEADER
I20260925 22:54:19.673403 21550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d68bf93b9e64696b13e23a8b6bcb797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d68bf93b9e64696b13e23a8b6bcb797" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38103 } }
I20260925 22:54:19.674017 21545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d68bf93b9e64696b13e23a8b6bcb797 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:19.675024 21551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d68bf93b9e64696b13e23a8b6bcb797 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d68bf93b9e64696b13e23a8b6bcb797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d68bf93b9e64696b13e23a8b6bcb797" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38103 } } }
I20260925 22:54:19.675141 21551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d68bf93b9e64696b13e23a8b6bcb797 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:19.675504 21552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d68bf93b9e64696b13e23a8b6bcb797 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d68bf93b9e64696b13e23a8b6bcb797. Latest consensus state: current_term: 1 leader_uuid: "7d68bf93b9e64696b13e23a8b6bcb797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d68bf93b9e64696b13e23a8b6bcb797" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38103 } } }
I20260925 22:54:19.675659 21552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d68bf93b9e64696b13e23a8b6bcb797 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:19.675973 21559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:54:19.677282 21548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:19.663411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37398 (local address 127.20.114.254:38103)
0925 22:54:19.663830 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:19.663844 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:19.663883 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:54:19.664546 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:19.664563 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:19.664995 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:19.665313 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:19.666082 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:19.667078 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:19.667920 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:19.668374 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:19.668555 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:19.671329 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:19.671335 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:19.671347 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:19.671371 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:19.673497 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:19.674040 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:19.674045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:19.674050 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:19.674185 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:19.674501 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:19.674504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:19.674506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:19.675049 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:19.675861 (+ 812us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:19.676017 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 22:54:19.676325 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":129,"threads_started":1}
I20260925 22:54:19.679131 21559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:54:19.684794 21559 catalog_manager.cc:1347] Generated new cluster ID: 3bc1aa2ac69c4e03be85c940953fb9a7
I20260925 22:54:19.684906 21559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:19.694181 21559 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:54:19.695056 21559 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:19.699640 21559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d68bf93b9e64696b13e23a8b6bcb797: Generated new TSK 0
I20260925 22:54:19.700270 21559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.114.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
I20260925 22:54:19.736609 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:38103
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:34597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:19.870620 21571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:19.870875 21571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:19.870926 21571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:19.875262 21571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:19.875327 21571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:19.875429 21571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:54:19.881178 21571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:34597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:38103
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:19.882220 21571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:19.883308 21571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:19.893117 21576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:19.893111 21577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:19.893124 21579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:19.893152 21571 server_base.cc:1034] running on GCE node
I20260925 22:54:19.893801 21571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:19.894438 21571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:19.895596 21571 hybrid_clock.cc:648] HybridClock initialized: now 1790376859895567 us; error 42 us; skew 500 ppm
Sep 25 22:54:19 dist-test-slave-x1lh krb5kdc[21464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376859, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:19.898289 21571 init.cc:377] Logged in from keytab as kudu/127.20.114.193@KRBTEST.COM (short username kudu)
I20260925 22:54:19.899245 21571 webserver.cc:460] Webserver started at http://127.20.114.193:39395/ using document root <none> and password file <none>
I20260925 22:54:19.899804 21571 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:19.899895 21571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:19.900132 21571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:19.902204 21571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "5218459026544bec832e1a92668b2dc7"
format_stamp: "Formatted at 2026-09-25 22:54:19 on dist-test-slave-x1lh"
server_key: "16a3e0d98c935ec0b6badb17896f0384"
server_key_iv: "b29569c9d1c441ddace786c33c4629a9"
server_key_version: "encryptionkey@0"
I20260925 22:54:19.902753 21571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "5218459026544bec832e1a92668b2dc7"
format_stamp: "Formatted at 2026-09-25 22:54:19 on dist-test-slave-x1lh"
server_key: "16a3e0d98c935ec0b6badb17896f0384"
server_key_iv: "b29569c9d1c441ddace786c33c4629a9"
server_key_version: "encryptionkey@0"
I20260925 22:54:19.906330 21571 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:54:19.908623 21586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:19.909858 21571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:19.910010 21571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "5218459026544bec832e1a92668b2dc7"
format_stamp: "Formatted at 2026-09-25 22:54:19 on dist-test-slave-x1lh"
server_key: "16a3e0d98c935ec0b6badb17896f0384"
server_key_iv: "b29569c9d1c441ddace786c33c4629a9"
server_key_version: "encryptionkey@0"
I20260925 22:54:19.910120 21571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:19.924423 21571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:19.927439 21571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:19.927646 21571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:19.928306 21571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:19.929211 21571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:19.929301 21571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:19.929370 21571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:19.929423 21571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:19.938278 21571 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:35267
I20260925 22:54:19.938306 21699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:35267 every 8 connection(s)
I20260925 22:54:19.939139 21571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:54:19.943668 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 21571
I20260925 22:54:19.943776 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:54:19.944038 20939 external_mini_cluster.cc:1428] Setting key 3c89caf3a6b974ea9c90f13da34529ae
Sep 25 22:54:19 dist-test-slave-x1lh krb5kdc[21464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376859, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.114.194@KRBTEST.COM; defaulting to no policy
I20260925 22:54:19.954449 21548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:19.940934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:41197 (local address 127.20.114.254:38103)
0925 22:54:19.941064 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:19.941067 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:19.941912 (+ 845us) server_negotiation.cc:407] Connection header received
0925 22:54:19.943083 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:19.943086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:19.943132 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:19.943205 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:19.944699 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:19.945481 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:19.947218 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:19.947647 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:19.947766 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:19.950633 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:19.950636 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:19.950638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:19.950660 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:19.952196 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:19.952813 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:19.952815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:19.952816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:19.952962 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:19.953349 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:19.953352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:19.953353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:19.953520 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:19.953591 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:19.954209 (+ 618us) server_negotiation.cc:299] Negotiation successful
0925 22:54:19.954329 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:54:19.955329 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:19.941338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:38103 (local address 127.20.114.193:41197)
0925 22:54:19.941749 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:19.941789 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:19.942881 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:19.943358 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:19.943364 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:19.943869 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:19.944543 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:19.944567 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:19.945610 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:19.945613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:19.947075 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:19.947089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:19.947790 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:19.947793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:19.947930 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:19.948534 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:19.948550 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:19.950462 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:19.952330 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:19.952336 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:19.952349 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:19.952718 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:19.953059 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:19.953064 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:19.953069 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:19.953246 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:19.953633 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:19.953642 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:19.953946 (+ 304us) client_negotiation.cc:769] Sending connection context
0925 22:54:19.954154 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 22:54:19.954515 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":96,"threads_started":1}
I20260925 22:54:19.956574 21700 heartbeater.cc:347] Connected to a master server at 127.20.114.254:38103
I20260925 22:54:19.956768 21700 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.194@KRBTEST.COM" created.
I20260925 22:54:19.957306 21700 heartbeater.cc:511] Master 127.20.114.254:38103 requested a full tablet report, sending...
I20260925 22:54:19.958973 21509 ts_manager.cc:194] Registered new tserver with Master: 5218459026544bec832e1a92668b2dc7 (127.20.114.193:35267)
I20260925 22:54:19.960165 21509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:41197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
I20260925 22:54:19.975852 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:38103
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:34597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:20.085685 21705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:20.085943 21705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:20.086022 21705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:20.090327 21705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:20.090389 21705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:20.090507 21705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:54:20.095896 21705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:34597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:38103
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:20.097040 21705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:20.098074 21705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:20.107697 21711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:20.107668 21713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:20.107666 21710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:20.108011 21705 server_base.cc:1034] running on GCE node
I20260925 22:54:20.108476 21705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:20.109143 21705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:20.110301 21705 hybrid_clock.cc:648] HybridClock initialized: now 1790376860110265 us; error 50 us; skew 500 ppm
Sep 25 22:54:20 dist-test-slave-x1lh krb5kdc[21464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376860, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:20.113111 21705 init.cc:377] Logged in from keytab as kudu/127.20.114.194@KRBTEST.COM (short username kudu)
I20260925 22:54:20.114050 21705 webserver.cc:460] Webserver started at http://127.20.114.194:35423/ using document root <none> and password file <none>
I20260925 22:54:20.114667 21705 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:20.114755 21705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:20.114971 21705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:20.117012 21705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "f447d16efced4cf2958a3f4631b051a9"
format_stamp: "Formatted at 2026-09-25 22:54:20 on dist-test-slave-x1lh"
server_key: "a458e75cffb4f48de815a908e558ec49"
server_key_iv: "bc0aa4020b4ee835b1eb08bb805d9178"
server_key_version: "encryptionkey@0"
I20260925 22:54:20.117578 21705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "f447d16efced4cf2958a3f4631b051a9"
format_stamp: "Formatted at 2026-09-25 22:54:20 on dist-test-slave-x1lh"
server_key: "a458e75cffb4f48de815a908e558ec49"
server_key_iv: "bc0aa4020b4ee835b1eb08bb805d9178"
server_key_version: "encryptionkey@0"
I20260925 22:54:20.121263 21705 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:54:20.123693 21720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:20.124961 21705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:20.125052 21705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "f447d16efced4cf2958a3f4631b051a9"
format_stamp: "Formatted at 2026-09-25 22:54:20 on dist-test-slave-x1lh"
server_key: "a458e75cffb4f48de815a908e558ec49"
server_key_iv: "bc0aa4020b4ee835b1eb08bb805d9178"
server_key_version: "encryptionkey@0"
I20260925 22:54:20.125145 21705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:20.141441 21705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:20.144165 21705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:20.144316 21705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:20.144826 21705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:20.145632 21705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:20.145674 21705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:20.145710 21705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:20.145732 21705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:20.154769 21705 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:42205
I20260925 22:54:20.154794 21833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:42205 every 8 connection(s)
I20260925 22:54:20.155655 21705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:54:20.162171 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 21705
I20260925 22:54:20.162302 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:54:20.162575 20939 external_mini_cluster.cc:1428] Setting key 8e72cd76d59edea7c23f8322cf72c663
Sep 25 22:54:20 dist-test-slave-x1lh krb5kdc[21464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376860, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:20.169224 21548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:20.157431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:41897 (local address 127.20.114.254:38103)
0925 22:54:20.157565 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:20.157569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:20.158309 (+ 740us) server_negotiation.cc:407] Connection header received
0925 22:54:20.159578 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:20.159582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:20.159643 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:20.159745 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:20.160903 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.161412 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.162634 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.162852 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.162975 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:20.165489 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:20.165496 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:20.165502 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:20.165532 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:20.167010 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:20.167653 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:20.167656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:20.167657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:20.167727 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:20.168118 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:20.168120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:20.168121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:20.168272 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:20.168343 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:20.169015 (+ 672us) server_negotiation.cc:299] Negotiation successful
0925 22:54:20.169111 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:54:20.170522 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:20.157730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:38103 (local address 127.20.114.194:41897)
0925 22:54:20.158155 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:20.158196 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:20.159346 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:20.159917 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:20.159924 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:20.160264 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:20.160723 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:20.160735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.161537 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.161541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:20.162516 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:20.162525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.163012 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.163016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:20.163186 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:20.163823 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:20.163839 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:20.165266 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:20.167128 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:20.167137 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:20.167154 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:20.167538 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:20.167821 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:20.167824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:20.167827 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:20.168020 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:20.168378 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:20.168386 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:20.168724 (+ 338us) client_negotiation.cc:769] Sending connection context
0925 22:54:20.168980 (+ 256us) client_negotiation.cc:240] Negotiation successful
0925 22:54:20.169460 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":114,"threads_started":1}
I20260925 22:54:20.171717 21834 heartbeater.cc:347] Connected to a master server at 127.20.114.254:38103
WARNING: no policy specified for kudu/127.20.114.195@KRBTEST.COM; defaulting to no policy
I20260925 22:54:20.171975 21834 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:20.172428 21834 heartbeater.cc:511] Master 127.20.114.254:38103 requested a full tablet report, sending...
I20260925 22:54:20.173556 21509 ts_manager.cc:194] Registered new tserver with Master: f447d16efced4cf2958a3f4631b051a9 (127.20.114.194:42205)
I20260925 22:54:20.174237 21509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:41897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
I20260925 22:54:20.192211 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:38103
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:34597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:20.301545 21839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:20.301797 21839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:20.301841 21839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:20.306161 21839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:20.306219 21839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:20.306329 21839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:54:20.311652 21839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:34597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:38103
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:20.312666 21839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:20.313795 21839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:20.323205 21844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:20.323222 21845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:20.323437 21839 server_base.cc:1034] running on GCE node
W20260925 22:54:20.323247 21847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:20.323884 21839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:20.324504 21839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:20.325690 21839 hybrid_clock.cc:648] HybridClock initialized: now 1790376860325675 us; error 26 us; skew 500 ppm
Sep 25 22:54:20 dist-test-slave-x1lh krb5kdc[21464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376860, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:20.328552 21839 init.cc:377] Logged in from keytab as kudu/127.20.114.195@KRBTEST.COM (short username kudu)
I20260925 22:54:20.329610 21839 webserver.cc:460] Webserver started at http://127.20.114.195:39879/ using document root <none> and password file <none>
I20260925 22:54:20.330165 21839 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:20.330256 21839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:20.330467 21839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:20.332422 21839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "3ef0b3eb299e4d95b97c5b6724b98d5a"
format_stamp: "Formatted at 2026-09-25 22:54:20 on dist-test-slave-x1lh"
server_key: "9aa5c6ea33a2e065a1c77122e9a97a48"
server_key_iv: "9d43a68889efb70c4440a11ad4b96b0b"
server_key_version: "encryptionkey@0"
I20260925 22:54:20.333037 21839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "3ef0b3eb299e4d95b97c5b6724b98d5a"
format_stamp: "Formatted at 2026-09-25 22:54:20 on dist-test-slave-x1lh"
server_key: "9aa5c6ea33a2e065a1c77122e9a97a48"
server_key_iv: "9d43a68889efb70c4440a11ad4b96b0b"
server_key_version: "encryptionkey@0"
I20260925 22:54:20.336546 21839 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:54:20.338903 21854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:20.340108 21839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:20.340238 21839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "3ef0b3eb299e4d95b97c5b6724b98d5a"
format_stamp: "Formatted at 2026-09-25 22:54:20 on dist-test-slave-x1lh"
server_key: "9aa5c6ea33a2e065a1c77122e9a97a48"
server_key_iv: "9d43a68889efb70c4440a11ad4b96b0b"
server_key_version: "encryptionkey@0"
I20260925 22:54:20.340363 21839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:20.352526 21839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:20.355747 21839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:20.355940 21839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:20.356487 21839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:20.357259 21839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:20.357331 21839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:20.357393 21839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:20.357446 21839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:20.365726 21839 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:35857
I20260925 22:54:20.365796 21967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:35857 every 8 connection(s)
I20260925 22:54:20.366616 21839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:54:20.368135 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 21839
I20260925 22:54:20.368278 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:54:20.368583 20939 external_mini_cluster.cc:1428] Setting key b08fecc01988ca4f8bed5b08c3835062
Sep 25 22:54:20 dist-test-slave-x1lh krb5kdc[21464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376860, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:20.379807 21548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:20.368416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:39241 (local address 127.20.114.254:38103)
0925 22:54:20.368576 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:20.368581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:20.369511 (+ 930us) server_negotiation.cc:407] Connection header received
0925 22:54:20.370898 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:20.370900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:20.370944 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:20.371008 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:20.372058 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.372562 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.373840 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.374051 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.374128 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:20.376604 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:20.376607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:20.376609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:20.376631 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:20.377927 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:20.378363 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:20.378366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:20.378367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:20.378443 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:20.378722 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:20.378724 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:20.378725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:20.378918 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:20.378992 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:20.379594 (+ 602us) server_negotiation.cc:299] Negotiation successful
0925 22:54:20.379689 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 22:54:20.380605 21970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:20.368813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:38103 (local address 127.20.114.195:39241)
0925 22:54:20.369360 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:20.369420 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:20.370706 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:20.371132 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:20.371139 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:20.371482 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:20.371938 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:20.371950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.372692 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.372695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:20.373733 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:20.373743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.374175 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.374178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:20.374307 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:20.375047 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:20.375063 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:20.376462 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:20.378029 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:20.378034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:20.378046 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:20.378281 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:20.378520 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:20.378522 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:20.378524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:20.378646 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:20.379058 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:20.379066 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:20.379385 (+ 319us) client_negotiation.cc:769] Sending connection context
0925 22:54:20.379543 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 22:54:20.379776 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":134,"threads_started":1}
I20260925 22:54:20.381771 21968 heartbeater.cc:347] Connected to a master server at 127.20.114.254:38103
I20260925 22:54:20.382014 21968 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:20.382560 21968 heartbeater.cc:511] Master 127.20.114.254:38103 requested a full tablet report, sending...
I20260925 22:54:20.383550 21508 ts_manager.cc:194] Registered new tserver with Master: 3ef0b3eb299e4d95b97c5b6724b98d5a (127.20.114.195:35857)
I20260925 22:54:20.384127 21508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:39241
I20260925 22:54:20.394407 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:54:20.405230 21548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:20.398149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37400 (local address 127.20.114.254:38103)
0925 22:54:20.398279 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:20.398283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:20.398459 (+ 176us) server_negotiation.cc:407] Connection header received
0925 22:54:20.398585 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:20.398587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:20.398630 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:20.398707 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:20.399387 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.399895 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.400662 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.400859 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.400957 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:20.402423 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:20.402426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:20.402428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:20.402448 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:20.403694 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:20.404080 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:20.404083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:20.404084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:20.404155 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:20.404372 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:20.404374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:20.404375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:20.404579 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:20.404678 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:20.405014 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 22:54:20.405111 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:54:20.410128 21508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:54:20.413168 21508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:54:20.427089 21979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:20.421126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:35267 (local address 127.0.0.1:46360)
0925 22:54:20.421489 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:20.421534 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:20.421652 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:20.422427 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:20.422435 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:20.422460 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:20.422628 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:20.422636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.424064 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.424067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:20.425529 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:20.425538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.426556 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.426559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:20.426669 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:20.426710 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 22:54:20.426811 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 22:54:20.426931 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":28,"thread_start_us":105,"threads_started":1}
I20260925 22:54:20.427253 21983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:20.421875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:35857 (local address 127.0.0.1:42150)
0925 22:54:20.422674 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:20.422685 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:20.422789 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:20.423273 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:20.423276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:20.423293 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:20.423382 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:20.423388 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.424565 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.424571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:20.425774 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:20.425783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.426789 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.426792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:20.426873 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:20.426905 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 22:54:20.426988 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 22:54:20.427057 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260925 22:54:20.427524 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:20.421295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46360 (local address 127.20.114.193:35267)
0925 22:54:20.421555 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:20.421577 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:20.421599 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:54:20.421795 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:20.421800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:20.422032 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:20.422197 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:20.422761 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.423920 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.425652 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.426435 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.426569 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:20.426701 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:20.427285 (+ 584us) server_negotiation.cc:299] Negotiation successful
0925 22:54:20.427383 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":105,"threads_started":1}
I20260925 22:54:20.427536 21982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:20.422048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42150 (local address 127.20.114.195:35857)
0925 22:54:20.422922 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:20.422933 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:20.422959 (+ 26us) server_negotiation.cc:407] Connection header received
0925 22:54:20.423014 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:20.423018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:20.423147 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:20.423254 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:20.423526 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.424439 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.425907 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.426690 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.426798 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:20.426913 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:20.427014 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 22:54:20.427102 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":177,"threads_started":1}
I20260925 22:54:20.427904 21981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:20.422241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:42205 (local address 127.0.0.1:40492)
0925 22:54:20.423439 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:20.423478 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:20.423575 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:20.423977 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:20.423980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:20.423997 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:20.424071 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:20.424077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.425446 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.425452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:20.426300 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:20.426308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.427519 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.427524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:20.427616 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:20.427644 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 22:54:20.427727 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 22:54:20.427795 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1145,"thread_start_us":69,"threads_started":1}
I20260925 22:54:20.428179 21984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:20.422643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40492 (local address 127.20.114.194:42205)
0925 22:54:20.423038 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:20.423049 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:20.423570 (+ 521us) server_negotiation.cc:407] Connection header received
0925 22:54:20.423674 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:20.423678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:20.423838 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:20.423982 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:20.424161 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.425315 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.426426 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.427219 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.427334 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:20.427456 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:20.427758 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 22:54:20.427858 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":104,"threads_started":1}
I20260925 22:54:20.430583 21634 tablet_service.cc:1467] Processing CreateTablet for tablet 9101f65f3ea8429dbeddaa4de581ac65 (DEFAULT_TABLE table=test-table [id=8e0fa3c21d414b19aa2a0cf91276d7c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:20.430583 21902 tablet_service.cc:1467] Processing CreateTablet for tablet 9101f65f3ea8429dbeddaa4de581ac65 (DEFAULT_TABLE table=test-table [id=8e0fa3c21d414b19aa2a0cf91276d7c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:20.430827 21768 tablet_service.cc:1467] Processing CreateTablet for tablet 9101f65f3ea8429dbeddaa4de581ac65 (DEFAULT_TABLE table=test-table [id=8e0fa3c21d414b19aa2a0cf91276d7c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:20.431699 21902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9101f65f3ea8429dbeddaa4de581ac65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:20.431691 21634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9101f65f3ea8429dbeddaa4de581ac65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:20.431886 21768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9101f65f3ea8429dbeddaa4de581ac65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:20.438014 21985 tablet_bootstrap.cc:492] T 9101f65f3ea8429dbeddaa4de581ac65 P f447d16efced4cf2958a3f4631b051a9: Bootstrap starting.
I20260925 22:54:20.439333 21987 tablet_bootstrap.cc:492] T 9101f65f3ea8429dbeddaa4de581ac65 P 3ef0b3eb299e4d95b97c5b6724b98d5a: Bootstrap starting.
I20260925 22:54:20.440642 21985 tablet_bootstrap.cc:654] T 9101f65f3ea8429dbeddaa4de581ac65 P f447d16efced4cf2958a3f4631b051a9: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:20.441192 21986 tablet_bootstrap.cc:492] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7: Bootstrap starting.
I20260925 22:54:20.441471 21985 log.cc:824] T 9101f65f3ea8429dbeddaa4de581ac65 P f447d16efced4cf2958a3f4631b051a9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:20.441635 21987 tablet_bootstrap.cc:654] T 9101f65f3ea8429dbeddaa4de581ac65 P 3ef0b3eb299e4d95b97c5b6724b98d5a: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:20.442458 21987 log.cc:824] T 9101f65f3ea8429dbeddaa4de581ac65 P 3ef0b3eb299e4d95b97c5b6724b98d5a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:20.443295 21986 tablet_bootstrap.cc:654] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:20.443929 21985 tablet_bootstrap.cc:492] T 9101f65f3ea8429dbeddaa4de581ac65 P f447d16efced4cf2958a3f4631b051a9: No bootstrap required, opened a new log
I20260925 22:54:20.444132 21985 ts_tablet_manager.cc:1397] T 9101f65f3ea8429dbeddaa4de581ac65 P f447d16efced4cf2958a3f4631b051a9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 22:54:20.444453 21986 log.cc:824] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:20.444577 21987 tablet_bootstrap.cc:492] T 9101f65f3ea8429dbeddaa4de581ac65 P 3ef0b3eb299e4d95b97c5b6724b98d5a: No bootstrap required, opened a new log
I20260925 22:54:20.444756 21987 ts_tablet_manager.cc:1397] T 9101f65f3ea8429dbeddaa4de581ac65 P 3ef0b3eb299e4d95b97c5b6724b98d5a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:54:20.446385 21986 tablet_bootstrap.cc:492] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7: No bootstrap required, opened a new log
I20260925 22:54:20.446573 21986 ts_tablet_manager.cc:1397] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:54:20.447508 21985 raft_consensus.cc:348] T 9101f65f3ea8429dbeddaa4de581ac65 P f447d16efced4cf2958a3f4631b051a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5218459026544bec832e1a92668b2dc7" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35267 } } peers { permanent_uuid: "3ef0b3eb299e4d95b97c5b6724b98d5a" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 35857 } } peers { permanent_uuid: "f447d16efced4cf2958a3f4631b051a9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42205 } }
I20260925 22:54:20.447754 21985 raft_consensus.cc:374] T 9101f65f3ea8429dbeddaa4de581ac65 P f447d16efced4cf2958a3f4631b051a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:20.447808 21985 raft_consensus.cc:729] T 9101f65f3ea8429dbeddaa4de581ac65 P f447d16efced4cf2958a3f4631b051a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f447d16efced4cf2958a3f4631b051a9, State: Initialized, Role: FOLLOWER
I20260925 22:54:20.448274 21985 consensus_queue.cc:260] T 9101f65f3ea8429dbeddaa4de581ac65 P f447d16efced4cf2958a3f4631b051a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5218459026544bec832e1a92668b2dc7" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35267 } } peers { permanent_uuid: "3ef0b3eb299e4d95b97c5b6724b98d5a" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 35857 } } peers { permanent_uuid: "f447d16efced4cf2958a3f4631b051a9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42205 } }
I20260925 22:54:20.448462 21987 raft_consensus.cc:348] T 9101f65f3ea8429dbeddaa4de581ac65 P 3ef0b3eb299e4d95b97c5b6724b98d5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5218459026544bec832e1a92668b2dc7" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35267 } } peers { permanent_uuid: "3ef0b3eb299e4d95b97c5b6724b98d5a" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 35857 } } peers { permanent_uuid: "f447d16efced4cf2958a3f4631b051a9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42205 } }
I20260925 22:54:20.448719 21987 raft_consensus.cc:374] T 9101f65f3ea8429dbeddaa4de581ac65 P 3ef0b3eb299e4d95b97c5b6724b98d5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:20.448773 21987 raft_consensus.cc:729] T 9101f65f3ea8429dbeddaa4de581ac65 P 3ef0b3eb299e4d95b97c5b6724b98d5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef0b3eb299e4d95b97c5b6724b98d5a, State: Initialized, Role: FOLLOWER
I20260925 22:54:20.449002 21834 heartbeater.cc:503] Master 127.20.114.254:38103 was elected leader, sending a full tablet report...
I20260925 22:54:20.449514 21987 consensus_queue.cc:260] T 9101f65f3ea8429dbeddaa4de581ac65 P 3ef0b3eb299e4d95b97c5b6724b98d5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5218459026544bec832e1a92668b2dc7" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35267 } } peers { permanent_uuid: "3ef0b3eb299e4d95b97c5b6724b98d5a" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 35857 } } peers { permanent_uuid: "f447d16efced4cf2958a3f4631b051a9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42205 } }
I20260925 22:54:20.449822 21985 ts_tablet_manager.cc:1428] T 9101f65f3ea8429dbeddaa4de581ac65 P f447d16efced4cf2958a3f4631b051a9: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260925 22:54:20.450408 21987 ts_tablet_manager.cc:1428] T 9101f65f3ea8429dbeddaa4de581ac65 P 3ef0b3eb299e4d95b97c5b6724b98d5a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:54:20.450346 21986 raft_consensus.cc:348] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5218459026544bec832e1a92668b2dc7" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35267 } } peers { permanent_uuid: "3ef0b3eb299e4d95b97c5b6724b98d5a" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 35857 } } peers { permanent_uuid: "f447d16efced4cf2958a3f4631b051a9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42205 } }
I20260925 22:54:20.450588 21986 raft_consensus.cc:374] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:20.450644 21986 raft_consensus.cc:729] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5218459026544bec832e1a92668b2dc7, State: Initialized, Role: FOLLOWER
I20260925 22:54:20.450675 21968 heartbeater.cc:503] Master 127.20.114.254:38103 was elected leader, sending a full tablet report...
I20260925 22:54:20.451053 21986 consensus_queue.cc:260] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5218459026544bec832e1a92668b2dc7" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35267 } } peers { permanent_uuid: "3ef0b3eb299e4d95b97c5b6724b98d5a" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 35857 } } peers { permanent_uuid: "f447d16efced4cf2958a3f4631b051a9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42205 } }
I20260925 22:54:20.451730 21700 heartbeater.cc:503] Master 127.20.114.254:38103 was elected leader, sending a full tablet report...
I20260925 22:54:20.451995 21986 ts_tablet_manager.cc:1428] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:54:20.522423 21993 raft_consensus.cc:482] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:54:20.522612 21993 raft_consensus.cc:504] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5218459026544bec832e1a92668b2dc7" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35267 } } peers { permanent_uuid: "3ef0b3eb299e4d95b97c5b6724b98d5a" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 35857 } } peers { permanent_uuid: "f447d16efced4cf2958a3f4631b051a9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42205 } }
I20260925 22:54:20.523645 21993 leader_election.cc:288] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ef0b3eb299e4d95b97c5b6724b98d5a (127.20.114.195:35857), f447d16efced4cf2958a3f4631b051a9 (127.20.114.194:42205)
I20260925 22:54:20.528868 21982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:20.524063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:57797 (local address 127.20.114.195:35857)
0925 22:54:20.524242 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:20.524247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:20.524409 (+ 162us) server_negotiation.cc:407] Connection header received
0925 22:54:20.524565 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:20.524571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:20.524624 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:20.524701 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:20.525179 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.525945 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.527467 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.528148 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.528286 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:20.528349 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:20.528636 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 22:54:20.528731 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:54:20.528901 21984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:20.524683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:49353 (local address 127.20.114.194:42205)
0925 22:54:20.524825 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:20.524830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:20.525041 (+ 211us) server_negotiation.cc:407] Connection header received
0925 22:54:20.525158 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:20.525162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:20.525213 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:20.525293 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:20.525564 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.526205 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.527645 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.528295 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.528400 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:20.528442 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:20.528691 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 22:54:20.528759 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 22:54:20.528957 21994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:20.524000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:35857 (local address 127.20.114.193:57797)
0925 22:54:20.524297 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:20.524310 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:20.524448 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:20.524909 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:20.524913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:20.524934 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:20.525069 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:20.525077 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.526064 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.526067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:20.527349 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:20.527359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.528306 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.528309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:20.528450 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:20.528521 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 22:54:20.528609 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 22:54:20.528694 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":16,"thread_start_us":178,"threads_started":1}
I20260925 22:54:20.529449 21995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:20.524567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:42205 (local address 127.20.114.193:49353)
0925 22:54:20.524946 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:20.524967 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:20.525057 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:20.525349 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:20.525352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:20.525366 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:20.525453 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:20.525457 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.526338 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.526341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:20.527470 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:20.527558 (+ 88us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.528394 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.528397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:20.528544 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:20.528577 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 22:54:20.529201 (+ 624us) client_negotiation.cc:240] Negotiation successful
0925 22:54:20.529314 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":17,"thread_start_us":122,"threads_started":1}
I20260925 22:54:20.529588 21922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9101f65f3ea8429dbeddaa4de581ac65" candidate_uuid: "5218459026544bec832e1a92668b2dc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef0b3eb299e4d95b97c5b6724b98d5a" is_pre_election: true
I20260925 22:54:20.530030 21922 raft_consensus.cc:2454] T 9101f65f3ea8429dbeddaa4de581ac65 P 3ef0b3eb299e4d95b97c5b6724b98d5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5218459026544bec832e1a92668b2dc7 in term 0.
I20260925 22:54:20.530143 21788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9101f65f3ea8429dbeddaa4de581ac65" candidate_uuid: "5218459026544bec832e1a92668b2dc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f447d16efced4cf2958a3f4631b051a9" is_pre_election: true
I20260925 22:54:20.530412 21788 raft_consensus.cc:2454] T 9101f65f3ea8429dbeddaa4de581ac65 P f447d16efced4cf2958a3f4631b051a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5218459026544bec832e1a92668b2dc7 in term 0.
I20260925 22:54:20.530527 21588 leader_election.cc:302] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ef0b3eb299e4d95b97c5b6724b98d5a, 5218459026544bec832e1a92668b2dc7; no voters:
I20260925 22:54:20.530831 21993 raft_consensus.cc:2781] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:54:20.530926 21993 raft_consensus.cc:482] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:54:20.530972 21993 raft_consensus.cc:3037] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:20.532019 21993 raft_consensus.cc:504] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5218459026544bec832e1a92668b2dc7" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35267 } } peers { permanent_uuid: "3ef0b3eb299e4d95b97c5b6724b98d5a" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 35857 } } peers { permanent_uuid: "f447d16efced4cf2958a3f4631b051a9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42205 } }
I20260925 22:54:20.532361 21993 leader_election.cc:288] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7 [CANDIDATE]: Term 1 election: Requested vote from peers 3ef0b3eb299e4d95b97c5b6724b98d5a (127.20.114.195:35857), f447d16efced4cf2958a3f4631b051a9 (127.20.114.194:42205)
I20260925 22:54:20.532737 21922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9101f65f3ea8429dbeddaa4de581ac65" candidate_uuid: "5218459026544bec832e1a92668b2dc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef0b3eb299e4d95b97c5b6724b98d5a"
I20260925 22:54:20.532922 21922 raft_consensus.cc:3037] T 9101f65f3ea8429dbeddaa4de581ac65 P 3ef0b3eb299e4d95b97c5b6724b98d5a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:20.533069 21788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9101f65f3ea8429dbeddaa4de581ac65" candidate_uuid: "5218459026544bec832e1a92668b2dc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f447d16efced4cf2958a3f4631b051a9"
I20260925 22:54:20.533208 21788 raft_consensus.cc:3037] T 9101f65f3ea8429dbeddaa4de581ac65 P f447d16efced4cf2958a3f4631b051a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:20.534338 21922 raft_consensus.cc:2454] T 9101f65f3ea8429dbeddaa4de581ac65 P 3ef0b3eb299e4d95b97c5b6724b98d5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5218459026544bec832e1a92668b2dc7 in term 1.
I20260925 22:54:20.534552 21788 raft_consensus.cc:2454] T 9101f65f3ea8429dbeddaa4de581ac65 P f447d16efced4cf2958a3f4631b051a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5218459026544bec832e1a92668b2dc7 in term 1.
I20260925 22:54:20.534642 21588 leader_election.cc:302] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ef0b3eb299e4d95b97c5b6724b98d5a, 5218459026544bec832e1a92668b2dc7; no voters:
I20260925 22:54:20.534795 21993 raft_consensus.cc:2781] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:20.534989 21993 raft_consensus.cc:686] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7 [term 1 LEADER]: Becoming Leader. State: Replica: 5218459026544bec832e1a92668b2dc7, State: Running, Role: LEADER
I20260925 22:54:20.535351 21993 consensus_queue.cc:237] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5218459026544bec832e1a92668b2dc7" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35267 } } peers { permanent_uuid: "3ef0b3eb299e4d95b97c5b6724b98d5a" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 35857 } } peers { permanent_uuid: "f447d16efced4cf2958a3f4631b051a9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42205 } }
I20260925 22:54:20.539062 21509 catalog_manager.cc:5495] T 9101f65f3ea8429dbeddaa4de581ac65 P 5218459026544bec832e1a92668b2dc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5218459026544bec832e1a92668b2dc7 (127.20.114.193). New cstate: current_term: 1 leader_uuid: "5218459026544bec832e1a92668b2dc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5218459026544bec832e1a92668b2dc7" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ef0b3eb299e4d95b97c5b6724b98d5a" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 35857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f447d16efced4cf2958a3f4631b051a9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42205 } health_report { overall_health: UNKNOWN } } }
Sep 25 22:54:20 dist-test-slave-x1lh krb5kdc[21464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.193@KRBTEST.COM
I20260925 22:54:20.580582 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:20.571604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46362 (local address 127.20.114.193:35267)
0925 22:54:20.571787 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:20.571792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:20.571933 (+ 141us) server_negotiation.cc:407] Connection header received
0925 22:54:20.572096 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:20.572099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:20.572151 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:20.572243 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:20.572980 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.573495 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.574437 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.574663 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.574777 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:20.577364 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:20.577378 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:20.577400 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:20.577423 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:20.578983 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:20.579429 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:20.579435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:20.579437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:20.579524 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:20.579754 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:20.579756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:20.579757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:20.580001 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:20.580154 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:20.580330 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 22:54:20.580458 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Sep 25 22:54:20 dist-test-slave-x1lh krb5kdc[21464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376860, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:54:20 dist-test-slave-x1lh krb5kdc[21464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376860, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.114.193@KRBTEST.COM
I20260925 22:54:20.607465 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:20.599215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46370 (local address 127.20.114.193:35267)
0925 22:54:20.599418 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:20.599426 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:20.599524 (+ 98us) server_negotiation.cc:407] Connection header received
0925 22:54:20.599707 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:20.599715 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:20.599758 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:20.599874 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:20.600670 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.601271 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.602095 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.602324 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.602439 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:20.604746 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:20.604749 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:20.604751 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:20.604774 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:20.606115 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:20.606479 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:20.606481 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:20.606483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:20.606550 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:20.606766 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:20.606768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:20.606769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:20.606923 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:20.607036 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:20.607255 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 22:54:20.607346 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 22:54:20.617633 21969 tablet.cc:2369] T 9101f65f3ea8429dbeddaa4de581ac65 P 3ef0b3eb299e4d95b97c5b6724b98d5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 22:54:20 dist-test-slave-x1lh krb5kdc[21464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:54:20 dist-test-slave-x1lh krb5kdc[21464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.193@KRBTEST.COM
I20260925 22:54:20.633013 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:20.624740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46378 (local address 127.20.114.193:35267)
0925 22:54:20.624968 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:20.624973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:20.625050 (+ 77us) server_negotiation.cc:407] Connection header received
0925 22:54:20.625215 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:20.625219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:20.625270 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:20.625347 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:20.626010 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.626516 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.627278 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:20.627489 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:20.627560 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:20.630125 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:20.630128 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:20.630130 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:20.630152 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:20.631571 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:20.631920 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:20.631922 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:20.631924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:20.631993 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:20.632227 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:20.632229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:20.632230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:20.632399 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:20.632512 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:20.632663 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 22:54:20.632754 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 22:54:20.634353 21614 tablet_service.cc:2908] Rejecting scan request for tablet 9101f65f3ea8429dbeddaa4de581ac65: Uninitialized: clean time has not yet been initialized
I20260925 22:54:20.636530 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 21571
I20260925 22:54:20.642323 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 21705
I20260925 22:54:20.647689 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 21839
I20260925 22:54:20.653429 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 21477
2026-09-25T22:54:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2334 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:54:20 dist-test-slave-x1lh krb5kdc[22010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:54:20 dist-test-slave-x1lh krb5kdc[22010](info): set up 2 sockets
Sep 25 22:54:20 dist-test-slave-x1lh krb5kdc[22010](info): commencing operation
krb5kdc: starting...
W20260925 22:54:21.704567 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:54:21 dist-test-slave-x1lh krb5kdc[22010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:54:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:54:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:54:21.825655 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:34211
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:34211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:21.933549 22023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:21.933854 22023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:21.933929 22023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:21.937934 22023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:21.938017 22023 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 22:54:21.938048 22023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:21.938090 22023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:21.938130 22023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:21.938184 22023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:21.943619 22023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:34211
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:34211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:21.944689 22023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:21.945765 22023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:21.953737 22031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:21.953755 22029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:21.953913 22023 server_base.cc:1034] running on GCE node
W20260925 22:54:21.953787 22028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:21.954587 22023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:21.955617 22023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:21.956769 22023 hybrid_clock.cc:648] HybridClock initialized: now 1790376861956756 us; error 25 us; skew 500 ppm
Sep 25 22:54:21 dist-test-slave-x1lh krb5kdc[22010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376861, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:21.959488 22023 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:54:21.960279 22023 webserver.cc:460] Webserver started at http://127.20.114.254:36403/ using document root <none> and password file <none>
I20260925 22:54:21.960808 22023 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:21.960911 22023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:21.961127 22023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:21.963063 22023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "9b40dd73e9ea4dd1828001acb5e9ff83"
format_stamp: "Formatted at 2026-09-25 22:54:21 on dist-test-slave-x1lh"
server_key: "a37d102b087b3e9c5486774a5cbdcae6"
server_key_iv: "b82fa952a81e0b1a265d889a76c5b7aa"
server_key_version: "encryptionkey@0"
I20260925 22:54:21.963622 22023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "9b40dd73e9ea4dd1828001acb5e9ff83"
format_stamp: "Formatted at 2026-09-25 22:54:21 on dist-test-slave-x1lh"
server_key: "a37d102b087b3e9c5486774a5cbdcae6"
server_key_iv: "b82fa952a81e0b1a265d889a76c5b7aa"
server_key_version: "encryptionkey@0"
I20260925 22:54:21.967128 22023 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 22:54:21.969478 22038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:21.970649 22023 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:21.970782 22023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "9b40dd73e9ea4dd1828001acb5e9ff83"
format_stamp: "Formatted at 2026-09-25 22:54:21 on dist-test-slave-x1lh"
server_key: "a37d102b087b3e9c5486774a5cbdcae6"
server_key_iv: "b82fa952a81e0b1a265d889a76c5b7aa"
server_key_version: "encryptionkey@0"
I20260925 22:54:21.970889 22023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:21.982422 22023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:21.985332 22023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:21.985524 22023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:21.992595 22023 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:34211
I20260925 22:54:21.992673 22100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:34211 every 8 connection(s)
I20260925 22:54:21.993589 22023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:54:21.996241 22101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:22.001085 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 22023
I20260925 22:54:22.001205 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:54:22.001451 20939 external_mini_cluster.cc:1428] Setting key 89573a01225114b67eac5d607697e0cc
I20260925 22:54:22.001799 22101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83: Bootstrap starting.
I20260925 22:54:22.004217 22101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:22.005178 22101 log.cc:824] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:22.007285 22101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83: No bootstrap required, opened a new log
Sep 25 22:54:22 dist-test-slave-x1lh krb5kdc[22010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:22.011747 22101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b40dd73e9ea4dd1828001acb5e9ff83" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34211 } }
I20260925 22:54:22.012007 22101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:22.012086 22101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b40dd73e9ea4dd1828001acb5e9ff83, State: Initialized, Role: FOLLOWER
I20260925 22:54:22.012562 22101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b40dd73e9ea4dd1828001acb5e9ff83" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34211 } }
I20260925 22:54:22.012735 22101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:54:22.012809 22101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:54:22.012987 22101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:22.014093 22101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b40dd73e9ea4dd1828001acb5e9ff83" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34211 } }
I20260925 22:54:22.014482 22101 leader_election.cc:302] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b40dd73e9ea4dd1828001acb5e9ff83; no voters:
I20260925 22:54:22.014876 22101 leader_election.cc:288] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:54:22.014948 22106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:22.015129 22106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83 [term 1 LEADER]: Becoming Leader. State: Replica: 9b40dd73e9ea4dd1828001acb5e9ff83, State: Running, Role: LEADER
I20260925 22:54:22.015442 22106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b40dd73e9ea4dd1828001acb5e9ff83" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34211 } }
I20260925 22:54:22.016198 22101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:22.017182 22108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b40dd73e9ea4dd1828001acb5e9ff83. Latest consensus state: current_term: 1 leader_uuid: "9b40dd73e9ea4dd1828001acb5e9ff83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b40dd73e9ea4dd1828001acb5e9ff83" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34211 } } }
I20260925 22:54:22.017294 22108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:22.017561 22107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b40dd73e9ea4dd1828001acb5e9ff83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b40dd73e9ea4dd1828001acb5e9ff83" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34211 } } }
I20260925 22:54:22.017634 22107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:22.017671 22115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:54:22.017759 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:22.002820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42240 (local address 127.20.114.254:34211)
0925 22:54:22.003235 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:22.003250 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:22.003292 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:54:22.003916 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:22.003933 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:22.004287 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:22.004620 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:22.005596 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.006709 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.007860 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.008450 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.008615 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:22.011546 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:22.011551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:22.011565 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:22.011592 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:22.013682 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:22.014242 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:22.014248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:22.014255 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:22.014400 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:22.014711 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:22.014713 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:22.014715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:22.015316 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:22.016110 (+ 794us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:22.016242 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 22:54:22.016605 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":141,"threads_started":1}
I20260925 22:54:22.020803 22115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:54:22.027161 22115 catalog_manager.cc:1347] Generated new cluster ID: f6ad3207de264ac68d1b9625c8873cea
I20260925 22:54:22.027243 22115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:22.048983 22115 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:54:22.050068 22115 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:22.062407 22115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83: Generated new TSK 0
I20260925 22:54:22.063181 22115 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.114.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
I20260925 22:54:22.097724 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:34211
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:22.205922 22127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:22.206214 22127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:22.206298 22127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:22.210294 22127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:22.210387 22127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:22.210457 22127 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:54:22.210536 22127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:54:22.215627 22127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.114.254:34211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:22.216750 22127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:22.217816 22127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:22.227137 22132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:22.227185 22127 server_base.cc:1034] running on GCE node
W20260925 22:54:22.227119 22135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:22.227129 22133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:22.227713 22127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:22.228359 22127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:22.229552 22127 hybrid_clock.cc:648] HybridClock initialized: now 1790376862229535 us; error 30 us; skew 500 ppm
Sep 25 22:54:22 dist-test-slave-x1lh krb5kdc[22010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376862, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:22.232196 22127 init.cc:377] Logged in from keytab as kudu/127.20.114.193@KRBTEST.COM (short username kudu)
I20260925 22:54:22.233080 22127 webserver.cc:460] Webserver started at http://127.20.114.193:44733/ using document root <none> and password file <none>
I20260925 22:54:22.233619 22127 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:22.233709 22127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:22.233934 22127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:22.235875 22127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "1518dd7291014e1b87d449c4ddfecc5d"
format_stamp: "Formatted at 2026-09-25 22:54:22 on dist-test-slave-x1lh"
server_key: "129cc1067aa49fcc5136b2edce34d7c9"
server_key_iv: "f524fd39252c602649923c7a59060726"
server_key_version: "encryptionkey@0"
I20260925 22:54:22.236426 22127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "1518dd7291014e1b87d449c4ddfecc5d"
format_stamp: "Formatted at 2026-09-25 22:54:22 on dist-test-slave-x1lh"
server_key: "129cc1067aa49fcc5136b2edce34d7c9"
server_key_iv: "f524fd39252c602649923c7a59060726"
server_key_version: "encryptionkey@0"
I20260925 22:54:22.240139 22127 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:54:22.242511 22142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:22.243639 22127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:54:22.243767 22127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "1518dd7291014e1b87d449c4ddfecc5d"
format_stamp: "Formatted at 2026-09-25 22:54:22 on dist-test-slave-x1lh"
server_key: "129cc1067aa49fcc5136b2edce34d7c9"
server_key_iv: "f524fd39252c602649923c7a59060726"
server_key_version: "encryptionkey@0"
I20260925 22:54:22.243875 22127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:22.256744 22127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:22.259934 22127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:22.260144 22127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:22.261608 22127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:22.261691 22127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:22.261780 22127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:22.261831 22127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:54:22 dist-test-slave-x1lh krb5kdc[22010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376862, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:22.272948 22127 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:37189
I20260925 22:54:22.272984 22257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:37189 every 8 connection(s)
I20260925 22:54:22.274156 22127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:54:22.276221 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:22.262620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:60355 (local address 127.20.114.254:34211)
0925 22:54:22.262744 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:22.262748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:22.263975 (+ 1227us) server_negotiation.cc:407] Connection header received
0925 22:54:22.265153 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:22.265157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:22.265228 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:22.265335 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:22.266624 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.267162 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.268982 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.269436 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.269579 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:22.272201 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:22.272207 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:22.272213 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:22.272249 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:22.273946 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:22.274601 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:22.274606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:22.274608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:22.274732 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:22.275169 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:22.275174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:22.275176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:22.275353 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:22.275457 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:22.275923 (+ 466us) server_negotiation.cc:299] Negotiation successful
0925 22:54:22.276038 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:54:22.277076 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:22.263213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:34211 (local address 127.20.114.193:60355)
0925 22:54:22.263728 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:22.263781 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:22.264907 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:22.265529 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:22.265536 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:22.265854 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:22.266461 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:22.266479 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.267286 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.267290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:22.268811 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:22.268822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.269615 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.269619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:22.269845 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:22.270477 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:22.270492 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:22.272028 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:22.274109 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:22.274118 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:22.274135 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:22.274494 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:22.274928 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:22.274931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:22.274933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:22.275072 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:22.275454 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:22.275461 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:22.275686 (+ 225us) client_negotiation.cc:769] Sending connection context
0925 22:54:22.275868 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 22:54:22.276177 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":139,"threads_started":1}
I20260925 22:54:22.278353 22258 heartbeater.cc:347] Connected to a master server at 127.20.114.254:34211
I20260925 22:54:22.278529 22258 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:22.279023 22258 heartbeater.cc:511] Master 127.20.114.254:34211 requested a full tablet report, sending...
I20260925 22:54:22.280459 22054 ts_manager.cc:194] Registered new tserver with Master: 1518dd7291014e1b87d449c4ddfecc5d (127.20.114.193:37189)
I20260925 22:54:22.281854 22054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:60355
I20260925 22:54:22.284456 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 22127
I20260925 22:54:22.284567 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:54:22.284843 20939 external_mini_cluster.cc:1428] Setting key 38b6eb2c508eb5e67b1c98c7e41efde3
I20260925 22:54:22.287961 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:22.280514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:51389 (local address 127.20.114.254:34211)
0925 22:54:22.280676 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:22.280680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:22.280911 (+ 231us) server_negotiation.cc:407] Connection header received
0925 22:54:22.281026 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:22.281029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:22.281071 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:22.281157 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:22.281587 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.282284 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.283159 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.283400 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.283506 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:22.284721 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:22.284724 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:22.284725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:22.284746 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:22.286340 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:22.286773 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:22.286778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:22.286779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:22.286859 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:22.287153 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:22.287156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:22.287158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:22.287324 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:22.287446 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:22.287701 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 22:54:22.287811 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 22:54:22.287971 22264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:22.280440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:34211 (local address 127.20.114.193:51389)
0925 22:54:22.280794 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:22.280804 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:22.280941 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:22.281212 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:22.281215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:22.281383 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:22.281477 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:22.281482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.282445 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.282448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:22.283044 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:22.283052 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.283518 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.283522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:22.283618 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:22.284121 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:22.284132 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:22.284588 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:22.286469 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:22.286475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:22.286477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:22.286675 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:22.286967 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:22.286970 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:22.286972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:22.287059 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:22.287438 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:22.287441 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:22.287537 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 22:54:22.287639 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 22:54:22.287774 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.20.114.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
I20260925 22:54:22.311672 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:34211
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:22.419409 22267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:22.419672 22267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:22.419744 22267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:22.423920 22267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:22.423974 22267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:22.424013 22267 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:54:22.424111 22267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:54:22.429253 22267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.114.254:34211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:22.430250 22267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:22.431224 22267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:22.440997 22273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:22.441008 22272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:22.441108 22267 server_base.cc:1034] running on GCE node
W20260925 22:54:22.440997 22275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:22.441720 22267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:22.442329 22267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:22.443516 22267 hybrid_clock.cc:648] HybridClock initialized: now 1790376862443503 us; error 24 us; skew 500 ppm
Sep 25 22:54:22 dist-test-slave-x1lh krb5kdc[22010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376862, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:22.446254 22267 init.cc:377] Logged in from keytab as kudu/127.20.114.194@KRBTEST.COM (short username kudu)
I20260925 22:54:22.447160 22267 webserver.cc:460] Webserver started at http://127.20.114.194:41675/ using document root <none> and password file <none>
I20260925 22:54:22.447768 22267 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:22.447853 22267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:22.448079 22267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:22.450235 22267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "845ed19308ab4670b3078ba5090d5108"
format_stamp: "Formatted at 2026-09-25 22:54:22 on dist-test-slave-x1lh"
server_key: "132a2891ba5f802aed5cf319c6a8c985"
server_key_iv: "3f3aacd28241d9e7e76a7e50b73aff45"
server_key_version: "encryptionkey@0"
I20260925 22:54:22.450788 22267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "845ed19308ab4670b3078ba5090d5108"
format_stamp: "Formatted at 2026-09-25 22:54:22 on dist-test-slave-x1lh"
server_key: "132a2891ba5f802aed5cf319c6a8c985"
server_key_iv: "3f3aacd28241d9e7e76a7e50b73aff45"
server_key_version: "encryptionkey@0"
I20260925 22:54:22.454411 22267 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 22:54:22.456709 22282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:22.457984 22267 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:54:22.458110 22267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "845ed19308ab4670b3078ba5090d5108"
format_stamp: "Formatted at 2026-09-25 22:54:22 on dist-test-slave-x1lh"
server_key: "132a2891ba5f802aed5cf319c6a8c985"
server_key_iv: "3f3aacd28241d9e7e76a7e50b73aff45"
server_key_version: "encryptionkey@0"
I20260925 22:54:22.458218 22267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:22.470606 22267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:22.473904 22267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:22.474171 22267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:22.475631 22267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:22.475677 22267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:22.475739 22267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:22.475770 22267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:54:22 dist-test-slave-x1lh krb5kdc[22010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376862, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:22.486562 22267 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:43233
I20260925 22:54:22.486696 22397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:43233 every 8 connection(s)
I20260925 22:54:22.487697 22267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:54:22.490484 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:22.476495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:47659 (local address 127.20.114.254:34211)
0925 22:54:22.476620 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:22.476623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:22.477658 (+ 1035us) server_negotiation.cc:407] Connection header received
0925 22:54:22.478953 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:22.478960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:22.479030 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:22.479135 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:22.480619 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.481566 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.483277 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.483679 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.483788 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:22.486612 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:22.486616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:22.486619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:22.486651 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:22.488298 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:22.488929 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:22.488933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:22.488934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:22.489025 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:22.489379 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:22.489382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:22.489384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:22.489575 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:22.489679 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:22.490180 (+ 501us) server_negotiation.cc:299] Negotiation successful
0925 22:54:22.490301 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 22:54:22.491281 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:22.477068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:34211 (local address 127.20.114.194:47659)
0925 22:54:22.477520 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:22.477560 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:22.478558 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:22.479345 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:22.479352 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:22.479829 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:22.480445 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:22.480466 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.481703 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.481707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:22.483120 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:22.483133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.483824 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.483827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:22.483959 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:22.484583 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:22.484607 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:22.486433 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:22.488436 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:22.488442 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:22.488459 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:22.488789 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:22.489124 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:22.489127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:22.489129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:22.489264 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:22.489689 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:22.489697 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:22.489933 (+ 236us) client_negotiation.cc:769] Sending connection context
0925 22:54:22.490123 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 22:54:22.490432 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":131,"threads_started":1}
I20260925 22:54:22.492396 22398 heartbeater.cc:347] Connected to a master server at 127.20.114.254:34211
I20260925 22:54:22.492544 22398 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:22.493111 22398 heartbeater.cc:511] Master 127.20.114.254:34211 requested a full tablet report, sending...
I20260925 22:54:22.494146 22054 ts_manager.cc:194] Registered new tserver with Master: 845ed19308ab4670b3078ba5090d5108 (127.20.114.194:43233)
I20260925 22:54:22.494957 22054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:47659
I20260925 22:54:22.497692 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 22267
I20260925 22:54:22.497814 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:54:22.498106 20939 external_mini_cluster.cc:1428] Setting key 390002bb9075aa00c776d933ec82e3af
I20260925 22:54:22.502266 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:22.494773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:52141 (local address 127.20.114.254:34211)
0925 22:54:22.494883 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:22.494886 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:22.495104 (+ 218us) server_negotiation.cc:407] Connection header received
0925 22:54:22.495229 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:22.495232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:22.495274 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:22.495357 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:22.495831 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.496320 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.497286 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.497506 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.497587 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:22.498804 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:22.498807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:22.498810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:22.498838 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:22.500501 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:22.501002 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:22.501008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:22.501013 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:22.501099 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:22.501482 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:22.501485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:22.501486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:22.501685 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:22.501760 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:22.502068 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 22:54:22.502156 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 22:54:22.502426 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:22.494705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:34211 (local address 127.20.114.194:52141)
0925 22:54:22.495017 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:22.495029 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:22.495142 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:22.495384 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:22.495386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:22.495590 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:22.495698 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:22.495705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.496488 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.496491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:22.497184 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:22.497192 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.497614 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.497617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:22.497722 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:22.498252 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:22.498268 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:22.498665 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:22.500610 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:22.500612 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:22.500614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:22.500839 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:22.501196 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:22.501201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:22.501205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:22.501315 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:22.501780 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:22.501790 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:22.501899 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 22:54:22.502014 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 22:54:22.502172 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.20.114.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
I20260925 22:54:22.524142 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:34211
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:22.632488 22407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:22.632723 22407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:22.632767 22407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:22.636704 22407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:22.636792 22407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:22.636907 22407 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:54:22.636997 22407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:54:22.642294 22407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.114.254:34211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:22.643329 22407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:22.644341 22407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:54:22.654013 22407 server_base.cc:1034] running on GCE node
W20260925 22:54:22.653957 22413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:22.653960 22412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:22.653954 22415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:22.654568 22407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:22.655151 22407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:22.656333 22407 hybrid_clock.cc:648] HybridClock initialized: now 1790376862656306 us; error 38 us; skew 500 ppm
Sep 25 22:54:22 dist-test-slave-x1lh krb5kdc[22010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376862, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:22.659179 22407 init.cc:377] Logged in from keytab as kudu/127.20.114.195@KRBTEST.COM (short username kudu)
I20260925 22:54:22.659969 22407 webserver.cc:460] Webserver started at http://127.20.114.195:42003/ using document root <none> and password file <none>
I20260925 22:54:22.660501 22407 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:22.660588 22407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:22.660920 22407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:22.663543 22407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "462806982da846268dcac68bd7c4f182"
format_stamp: "Formatted at 2026-09-25 22:54:22 on dist-test-slave-x1lh"
server_key: "4ba61a4da04cb7613b3d0ee750ada56f"
server_key_iv: "02ea8927ffcaa513579af679d506c1c0"
server_key_version: "encryptionkey@0"
I20260925 22:54:22.664139 22407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "462806982da846268dcac68bd7c4f182"
format_stamp: "Formatted at 2026-09-25 22:54:22 on dist-test-slave-x1lh"
server_key: "4ba61a4da04cb7613b3d0ee750ada56f"
server_key_iv: "02ea8927ffcaa513579af679d506c1c0"
server_key_version: "encryptionkey@0"
I20260925 22:54:22.668663 22407 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260925 22:54:22.671227 22422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:22.672392 22407 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:22.672487 22407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "462806982da846268dcac68bd7c4f182"
format_stamp: "Formatted at 2026-09-25 22:54:22 on dist-test-slave-x1lh"
server_key: "4ba61a4da04cb7613b3d0ee750ada56f"
server_key_iv: "02ea8927ffcaa513579af679d506c1c0"
server_key_version: "encryptionkey@0"
I20260925 22:54:22.672565 22407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:22.689886 22407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:22.693259 22407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:22.693410 22407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:22.694738 22407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:22.694787 22407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:22.694826 22407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:22.694890 22407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:54:22 dist-test-slave-x1lh krb5kdc[22010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376862, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:22.705385 22407 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:42563
I20260925 22:54:22.705476 22537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:42563 every 8 connection(s)
I20260925 22:54:22.706293 22407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:54:22.709478 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:22.695693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:58599 (local address 127.20.114.254:34211)
0925 22:54:22.695820 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:22.695824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:22.696889 (+ 1065us) server_negotiation.cc:407] Connection header received
0925 22:54:22.698003 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:22.698006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:22.698066 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:22.698153 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:22.699797 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.700665 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.702519 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.702732 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.702815 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:22.705794 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:22.705797 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:22.705799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:22.705821 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:22.707389 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:22.707959 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:22.707962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:22.707964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:22.708045 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:22.708372 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:22.708375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:22.708377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:22.708556 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:22.708655 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:22.709177 (+ 522us) server_negotiation.cc:299] Negotiation successful
0925 22:54:22.709310 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:54:22.710000 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 22407
I20260925 22:54:22.710134 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:54:22.710222 22431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:22.696247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:34211 (local address 127.20.114.195:58599)
0925 22:54:22.696717 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:22.696758 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:22.697719 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:22.698403 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:22.698417 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:22.698886 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:22.699596 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:22.699618 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.700807 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.700814 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:22.702402 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:22.702412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.702890 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.702896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:22.703104 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:22.703788 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:22.703814 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:22.705634 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:22.707507 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:22.707515 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:22.707533 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:22.707853 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:22.708133 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:22.708137 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:22.708140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:22.708271 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:22.708663 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:22.708672 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:22.708944 (+ 272us) client_negotiation.cc:769] Sending connection context
0925 22:54:22.709123 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 22:54:22.709435 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":135,"threads_started":1}
I20260925 22:54:22.710460 20939 external_mini_cluster.cc:1428] Setting key 618c30678a669d4b111724cd7a878f45
I20260925 22:54:22.711582 22538 heartbeater.cc:347] Connected to a master server at 127.20.114.254:34211
I20260925 22:54:22.711719 22538 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:22.712236 22538 heartbeater.cc:511] Master 127.20.114.254:34211 requested a full tablet report, sending...
I20260925 22:54:22.713201 22055 ts_manager.cc:194] Registered new tserver with Master: 462806982da846268dcac68bd7c4f182 (127.20.114.195:42563)
I20260925 22:54:22.713835 22055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:58599
I20260925 22:54:22.720244 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:22.713717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:40715 (local address 127.20.114.254:34211)
0925 22:54:22.714140 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:22.714144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:22.714158 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:54:22.714188 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:22.714191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:22.714232 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:22.714288 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:22.714712 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.715414 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.716247 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.716454 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.716530 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:22.717468 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:22.717471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:22.717474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:22.717496 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:22.718766 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:22.719165 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:22.719169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:22.719171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:22.719245 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:22.719507 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:22.719512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:22.719513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:22.719669 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:22.719772 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:22.720013 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 22:54:22.720113 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358}
I20260925 22:54:22.720257 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:22.713653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:34211 (local address 127.20.114.195:40715)
0925 22:54:22.713913 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:22.713928 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:22.714007 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:22.714328 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:22.714332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:22.714514 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:22.714607 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:22.714612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.715533 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.715535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:22.716155 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:22.716163 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.716544 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.716547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:22.716612 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:22.717052 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:22.717063 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:22.717355 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:22.718869 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:22.718873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:22.718875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:22.719066 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:22.719332 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:22.719337 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:22.719338 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:22.719421 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:22.719761 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:22.719764 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:22.719846 (+ 82us) client_negotiation.cc:769] Sending connection context
0925 22:54:22.719962 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 22:54:22.720087 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":103,"threads_started":1}
I20260925 22:54:22.723802 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:54:22 dist-test-slave-x1lh krb5kdc[22010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376862, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:54:22 dist-test-slave-x1lh krb5kdc[22010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376862, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:22.747815 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:22.740188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42256 (local address 127.20.114.254:34211)
0925 22:54:22.740301 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:22.740304 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:22.740347 (+ 43us) server_negotiation.cc:407] Connection header received
0925 22:54:22.740382 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:22.740384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:22.740419 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:22.740480 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:22.741215 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.741769 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.742557 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.742758 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.742839 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:22.745066 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:22.745069 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:22.745071 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:22.745092 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:22.746326 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:22.746698 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:22.746701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:22.746702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:22.746771 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:22.747014 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:22.747016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:22.747017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:22.747239 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:22.747333 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:22.747532 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 22:54:22.747617 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:54:22.750522 22055 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:54:22.753558 22055 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:54:22.767210 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:22.761515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:37189 (local address 127.0.0.1:42284)
0925 22:54:22.761922 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:22.761977 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:22.762094 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:22.762687 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:22.762692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:22.762715 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:22.762880 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:22.762887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.764309 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.764312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:22.765638 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:22.765648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.766673 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.766676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:22.766787 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:22.766823 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 22:54:22.766958 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 22:54:22.767072 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":89,"threads_started":1}
I20260925 22:54:22.767226 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:22.761695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42284 (local address 127.20.114.193:37189)
0925 22:54:22.762065 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:22.762080 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:22.762106 (+ 26us) server_negotiation.cc:407] Connection header received
0925 22:54:22.762259 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:22.762266 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:22.762492 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:22.762691 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:22.763003 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.764151 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.765748 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.766553 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.766706 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:22.766842 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:22.766986 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 22:54:22.767072 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":83,"threads_started":1}
I20260925 22:54:22.768054 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:22.762276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:42563 (local address 127.0.0.1:35882)
0925 22:54:22.763190 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:22.763222 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:22.763311 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:22.763762 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:22.763765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:22.763782 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:22.763874 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:22.763882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.764949 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.764953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:22.765903 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:22.765912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.766952 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.766956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:22.767765 (+ 809us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:22.767791 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:54:22.767840 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 22:54:22.767906 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260925 22:54:22.768479 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:22.761842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:43233 (local address 127.0.0.1:48890)
0925 22:54:22.762412 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:22.762427 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:22.762508 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:22.764017 (+ 1509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:22.764020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:22.764036 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:22.764120 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:22.764125 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.765479 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.765482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:22.766575 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:22.766585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.768111 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.768115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:22.768211 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:22.768241 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 22:54:22.768306 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 22:54:22.768368 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":92,"threads_started":1}
I20260925 22:54:22.768548 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:22.762310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48890 (local address 127.20.114.194:43233)
0925 22:54:22.763577 (+ 1267us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:22.763591 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:22.763614 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:54:22.763685 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:22.763692 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:22.763867 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:22.764041 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:22.764208 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.765353 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.766711 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.767988 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.768128 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:22.768271 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:22.768329 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 22:54:22.768417 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1160,"thread_start_us":110,"threads_started":1}
I20260925 22:54:22.768980 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:22.762700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35882 (local address 127.20.114.195:42563)
0925 22:54:22.763040 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:22.763053 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:22.763313 (+ 260us) server_negotiation.cc:407] Connection header received
0925 22:54:22.763431 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:22.763436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:22.763608 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:22.763778 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:22.763984 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.764795 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.766036 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:22.766827 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:22.768439 (+ 1612us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:22.768695 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:22.768742 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 22:54:22.768825 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":118,"threads_started":1}
I20260925 22:54:22.770530 22192 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2b21dee9524cb9b97d88ca45d10508 (DEFAULT_TABLE table=test-table [id=9f8d448d072c4cf48ed2f93745b42aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:22.771128 22332 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2b21dee9524cb9b97d88ca45d10508 (DEFAULT_TABLE table=test-table [id=9f8d448d072c4cf48ed2f93745b42aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:22.771687 22192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2b21dee9524cb9b97d88ca45d10508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:22.772110 22332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2b21dee9524cb9b97d88ca45d10508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:22.775223 22472 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2b21dee9524cb9b97d88ca45d10508 (DEFAULT_TABLE table=test-table [id=9f8d448d072c4cf48ed2f93745b42aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:22.776192 22472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2b21dee9524cb9b97d88ca45d10508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:22.780205 22561 tablet_bootstrap.cc:492] T 8a2b21dee9524cb9b97d88ca45d10508 P 845ed19308ab4670b3078ba5090d5108: Bootstrap starting.
I20260925 22:54:22.781200 22562 tablet_bootstrap.cc:492] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d: Bootstrap starting.
I20260925 22:54:22.782442 22561 tablet_bootstrap.cc:654] T 8a2b21dee9524cb9b97d88ca45d10508 P 845ed19308ab4670b3078ba5090d5108: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:22.782804 22563 tablet_bootstrap.cc:492] T 8a2b21dee9524cb9b97d88ca45d10508 P 462806982da846268dcac68bd7c4f182: Bootstrap starting.
I20260925 22:54:22.783124 22562 tablet_bootstrap.cc:654] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:22.783274 22561 log.cc:824] T 8a2b21dee9524cb9b97d88ca45d10508 P 845ed19308ab4670b3078ba5090d5108: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:22.783895 22562 log.cc:824] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:22.785048 22563 tablet_bootstrap.cc:654] T 8a2b21dee9524cb9b97d88ca45d10508 P 462806982da846268dcac68bd7c4f182: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:22.785115 22561 tablet_bootstrap.cc:492] T 8a2b21dee9524cb9b97d88ca45d10508 P 845ed19308ab4670b3078ba5090d5108: No bootstrap required, opened a new log
I20260925 22:54:22.785313 22561 ts_tablet_manager.cc:1397] T 8a2b21dee9524cb9b97d88ca45d10508 P 845ed19308ab4670b3078ba5090d5108: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:54:22.785806 22563 log.cc:824] T 8a2b21dee9524cb9b97d88ca45d10508 P 462806982da846268dcac68bd7c4f182: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:22.786530 22562 tablet_bootstrap.cc:492] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d: No bootstrap required, opened a new log
I20260925 22:54:22.786708 22562 ts_tablet_manager.cc:1397] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260925 22:54:22.787392 22563 tablet_bootstrap.cc:492] T 8a2b21dee9524cb9b97d88ca45d10508 P 462806982da846268dcac68bd7c4f182: No bootstrap required, opened a new log
I20260925 22:54:22.787583 22563 ts_tablet_manager.cc:1397] T 8a2b21dee9524cb9b97d88ca45d10508 P 462806982da846268dcac68bd7c4f182: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260925 22:54:22.789178 22561 raft_consensus.cc:348] T 8a2b21dee9524cb9b97d88ca45d10508 P 845ed19308ab4670b3078ba5090d5108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } }
I20260925 22:54:22.789471 22561 raft_consensus.cc:374] T 8a2b21dee9524cb9b97d88ca45d10508 P 845ed19308ab4670b3078ba5090d5108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:22.789527 22561 raft_consensus.cc:729] T 8a2b21dee9524cb9b97d88ca45d10508 P 845ed19308ab4670b3078ba5090d5108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 845ed19308ab4670b3078ba5090d5108, State: Initialized, Role: FOLLOWER
I20260925 22:54:22.789961 22561 consensus_queue.cc:260] T 8a2b21dee9524cb9b97d88ca45d10508 P 845ed19308ab4670b3078ba5090d5108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } }
I20260925 22:54:22.790396 22562 raft_consensus.cc:348] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } }
I20260925 22:54:22.790638 22562 raft_consensus.cc:374] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:22.790681 22562 raft_consensus.cc:729] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1518dd7291014e1b87d449c4ddfecc5d, State: Initialized, Role: FOLLOWER
I20260925 22:54:22.790939 22561 ts_tablet_manager.cc:1428] T 8a2b21dee9524cb9b97d88ca45d10508 P 845ed19308ab4670b3078ba5090d5108: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:54:22.790769 22563 raft_consensus.cc:348] T 8a2b21dee9524cb9b97d88ca45d10508 P 462806982da846268dcac68bd7c4f182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } }
I20260925 22:54:22.791070 22563 raft_consensus.cc:374] T 8a2b21dee9524cb9b97d88ca45d10508 P 462806982da846268dcac68bd7c4f182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:22.791131 22563 raft_consensus.cc:729] T 8a2b21dee9524cb9b97d88ca45d10508 P 462806982da846268dcac68bd7c4f182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 462806982da846268dcac68bd7c4f182, State: Initialized, Role: FOLLOWER
I20260925 22:54:22.791083 22562 consensus_queue.cc:260] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } }
I20260925 22:54:22.791280 22398 heartbeater.cc:503] Master 127.20.114.254:34211 was elected leader, sending a full tablet report...
I20260925 22:54:22.791551 22563 consensus_queue.cc:260] T 8a2b21dee9524cb9b97d88ca45d10508 P 462806982da846268dcac68bd7c4f182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } }
I20260925 22:54:22.791923 22562 ts_tablet_manager.cc:1428] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:54:22.792222 22258 heartbeater.cc:503] Master 127.20.114.254:34211 was elected leader, sending a full tablet report...
I20260925 22:54:22.792548 22538 heartbeater.cc:503] Master 127.20.114.254:34211 was elected leader, sending a full tablet report...
I20260925 22:54:22.793516 22563 ts_tablet_manager.cc:1428] T 8a2b21dee9524cb9b97d88ca45d10508 P 462806982da846268dcac68bd7c4f182: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
W20260925 22:54:22.957255 22539 tablet.cc:2369] T 8a2b21dee9524cb9b97d88ca45d10508 P 462806982da846268dcac68bd7c4f182: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:54:22.988837 22399 tablet.cc:2369] T 8a2b21dee9524cb9b97d88ca45d10508 P 845ed19308ab4670b3078ba5090d5108: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:54:23.025539 22259 tablet.cc:2369] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:54:23.089274 22568 raft_consensus.cc:482] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:54:23.089551 22568 raft_consensus.cc:504] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } }
I20260925 22:54:23.092422 22568 leader_election.cc:288] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 462806982da846268dcac68bd7c4f182 (127.20.114.195:42563), 845ed19308ab4670b3078ba5090d5108 (127.20.114.194:43233)
I20260925 22:54:23.096014 22570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:23.091469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:42563 (local address 127.20.114.193:51771)
0925 22:54:23.091735 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:23.091753 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:23.091888 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:23.092286 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:23.092289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:23.092311 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:23.092441 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:23.092447 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.093612 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.093615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:23.094914 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:23.094924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.095566 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.095570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:23.095666 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:23.095693 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:54:23.095792 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 22:54:23.095858 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":81,"threads_started":1}
I20260925 22:54:23.096001 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:23.091597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:51771 (local address 127.20.114.195:42563)
0925 22:54:23.091734 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:23.091738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:23.091823 (+ 85us) server_negotiation.cc:407] Connection header received
0925 22:54:23.092019 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:23.092024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:23.092071 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:23.092118 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:23.092654 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.093478 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.095043 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.095451 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.095578 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:23.095631 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:23.095810 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 22:54:23.095878 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:54:23.096930 22492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2b21dee9524cb9b97d88ca45d10508" candidate_uuid: "1518dd7291014e1b87d449c4ddfecc5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "462806982da846268dcac68bd7c4f182" is_pre_election: true
I20260925 22:54:23.097332 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:23.092491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:56579 (local address 127.20.114.194:43233)
0925 22:54:23.092628 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:23.092633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:23.092645 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:54:23.092693 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:23.092695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:23.092740 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:23.092816 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:23.093131 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.094009 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.095702 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.096445 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.096560 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:23.096623 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:23.097072 (+ 449us) server_negotiation.cc:299] Negotiation successful
0925 22:54:23.097176 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:54:23.097309 22492 raft_consensus.cc:2454] T 8a2b21dee9524cb9b97d88ca45d10508 P 462806982da846268dcac68bd7c4f182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1518dd7291014e1b87d449c4ddfecc5d in term 0.
I20260925 22:54:23.097759 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:23.092092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:43233 (local address 127.20.114.193:56579)
0925 22:54:23.092426 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:23.092440 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:23.092537 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:23.092841 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:23.092843 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:23.092860 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:23.093014 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:23.093019 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.094155 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.094160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:23.095538 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:23.095551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.096726 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.096730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:23.096906 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:23.096941 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 22:54:23.097474 (+ 533us) client_negotiation.cc:240] Negotiation successful
0925 22:54:23.097568 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":112,"threads_started":1}
I20260925 22:54:23.097862 22143 leader_election.cc:302] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1518dd7291014e1b87d449c4ddfecc5d, 462806982da846268dcac68bd7c4f182; no voters:
I20260925 22:54:23.098158 22568 raft_consensus.cc:2781] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:54:23.098260 22568 raft_consensus.cc:482] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:54:23.098305 22568 raft_consensus.cc:3037] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:23.098404 22352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2b21dee9524cb9b97d88ca45d10508" candidate_uuid: "1518dd7291014e1b87d449c4ddfecc5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "845ed19308ab4670b3078ba5090d5108" is_pre_election: true
I20260925 22:54:23.098687 22352 raft_consensus.cc:2454] T 8a2b21dee9524cb9b97d88ca45d10508 P 845ed19308ab4670b3078ba5090d5108 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1518dd7291014e1b87d449c4ddfecc5d in term 0.
I20260925 22:54:23.099797 22568 raft_consensus.cc:504] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } }
I20260925 22:54:23.100263 22568 leader_election.cc:288] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d [CANDIDATE]: Term 1 election: Requested vote from peers 462806982da846268dcac68bd7c4f182 (127.20.114.195:42563), 845ed19308ab4670b3078ba5090d5108 (127.20.114.194:43233)
I20260925 22:54:23.100528 22492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2b21dee9524cb9b97d88ca45d10508" candidate_uuid: "1518dd7291014e1b87d449c4ddfecc5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "462806982da846268dcac68bd7c4f182"
I20260925 22:54:23.100586 22352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2b21dee9524cb9b97d88ca45d10508" candidate_uuid: "1518dd7291014e1b87d449c4ddfecc5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "845ed19308ab4670b3078ba5090d5108"
I20260925 22:54:23.100668 22492 raft_consensus.cc:3037] T 8a2b21dee9524cb9b97d88ca45d10508 P 462806982da846268dcac68bd7c4f182 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:23.100723 22352 raft_consensus.cc:3037] T 8a2b21dee9524cb9b97d88ca45d10508 P 845ed19308ab4670b3078ba5090d5108 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:23.102276 22352 raft_consensus.cc:2454] T 8a2b21dee9524cb9b97d88ca45d10508 P 845ed19308ab4670b3078ba5090d5108 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1518dd7291014e1b87d449c4ddfecc5d in term 1.
I20260925 22:54:23.102538 22492 raft_consensus.cc:2454] T 8a2b21dee9524cb9b97d88ca45d10508 P 462806982da846268dcac68bd7c4f182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1518dd7291014e1b87d449c4ddfecc5d in term 1.
I20260925 22:54:23.102604 22145 leader_election.cc:302] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1518dd7291014e1b87d449c4ddfecc5d, 845ed19308ab4670b3078ba5090d5108; no voters:
I20260925 22:54:23.102794 22568 raft_consensus.cc:2781] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:23.103034 22568 raft_consensus.cc:686] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d [term 1 LEADER]: Becoming Leader. State: Replica: 1518dd7291014e1b87d449c4ddfecc5d, State: Running, Role: LEADER
I20260925 22:54:23.103382 22568 consensus_queue.cc:237] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } }
I20260925 22:54:23.107350 22054 catalog_manager.cc:5495] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 1518dd7291014e1b87d449c4ddfecc5d (127.20.114.193). New cstate: current_term: 1 leader_uuid: "1518dd7291014e1b87d449c4ddfecc5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } health_report { overall_health: UNKNOWN } } }
I20260925 22:54:23.133698 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:23.129975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42290 (local address 127.20.114.193:37189)
0925 22:54:23.130117 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:23.130121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:23.130135 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:54:23.130342 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:23.130345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:23.130415 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:23.130524 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:23.130787 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.131308 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.132212 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.132441 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.132550 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:23.132890 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:23.133291 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:23.133425 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:23.133497 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 22:54:23.133551 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:54:23.139253 22352 raft_consensus.cc:1261] T 8a2b21dee9524cb9b97d88ca45d10508 P 845ed19308ab4670b3078ba5090d5108 [term 1 FOLLOWER]: Refusing update from remote peer 1518dd7291014e1b87d449c4ddfecc5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:23.139253 22492 raft_consensus.cc:1261] T 8a2b21dee9524cb9b97d88ca45d10508 P 462806982da846268dcac68bd7c4f182 [term 1 FOLLOWER]: Refusing update from remote peer 1518dd7291014e1b87d449c4ddfecc5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:23.139945 22572 consensus_queue.cc:1035] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:23.140132 22568 consensus_queue.cc:1035] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:23.159288 22050 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:23.159487 22050 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:23.161983 22050 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83: Sending DeleteTablet for 3 replicas of tablet 8a2b21dee9524cb9b97d88ca45d10508
I20260925 22:54:23.162801 22472 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a2b21dee9524cb9b97d88ca45d10508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:23 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:35882
I20260925 22:54:23.162787 22332 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a2b21dee9524cb9b97d88ca45d10508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:23 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:48890
I20260925 22:54:23.163022 22192 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a2b21dee9524cb9b97d88ca45d10508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:23 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:42284
I20260925 22:54:23.163287 22585 tablet_replica.cc:330] T 8a2b21dee9524cb9b97d88ca45d10508 P 845ed19308ab4670b3078ba5090d5108: stopping tablet replica
I20260925 22:54:23.163571 22586 tablet_replica.cc:330] T 8a2b21dee9524cb9b97d88ca45d10508 P 462806982da846268dcac68bd7c4f182: stopping tablet replica
I20260925 22:54:23.163602 22585 raft_consensus.cc:2229] T 8a2b21dee9524cb9b97d88ca45d10508 P 845ed19308ab4670b3078ba5090d5108 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:23.163657 22050 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 22:54:23.163847 22586 raft_consensus.cc:2229] T 8a2b21dee9524cb9b97d88ca45d10508 P 462806982da846268dcac68bd7c4f182 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:23.163892 22585 raft_consensus.cc:2258] T 8a2b21dee9524cb9b97d88ca45d10508 P 845ed19308ab4670b3078ba5090d5108 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 22:54:23.164031 22050 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:54:23.164103 22586 raft_consensus.cc:2258] T 8a2b21dee9524cb9b97d88ca45d10508 P 462806982da846268dcac68bd7c4f182 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:23.164224 22587 tablet_replica.cc:330] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d: stopping tablet replica
I20260925 22:54:23.164566 22587 raft_consensus.cc:2229] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:54:23.164937 22587 raft_consensus.cc:2258] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:23.165738 22585 ts_tablet_manager.cc:1905] T 8a2b21dee9524cb9b97d88ca45d10508 P 845ed19308ab4670b3078ba5090d5108: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:23.166121 22587 ts_tablet_manager.cc:1905] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:23.168409 22585 ts_tablet_manager.cc:1918] T 8a2b21dee9524cb9b97d88ca45d10508 P 845ed19308ab4670b3078ba5090d5108: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:54:23.168489 22585 log.cc:1196] T 8a2b21dee9524cb9b97d88ca45d10508 P 845ed19308ab4670b3078ba5090d5108: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/8a2b21dee9524cb9b97d88ca45d10508
I20260925 22:54:23.168772 22585 ts_tablet_manager.cc:1939] T 8a2b21dee9524cb9b97d88ca45d10508 P 845ed19308ab4670b3078ba5090d5108: Deleting consensus metadata
I20260925 22:54:23.168807 22587 ts_tablet_manager.cc:1918] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:54:23.168926 22587 log.cc:1196] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-0/wal/wals/8a2b21dee9524cb9b97d88ca45d10508
I20260925 22:54:23.169242 22587 ts_tablet_manager.cc:1939] T 8a2b21dee9524cb9b97d88ca45d10508 P 1518dd7291014e1b87d449c4ddfecc5d: Deleting consensus metadata
I20260925 22:54:23.169677 22192 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1497f5300c47e3866d1ffeaaf47c43 (DEFAULT_TABLE table=test-table [id=97fc0548c09047afb8199b025714bb97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:23.169677 22472 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1497f5300c47e3866d1ffeaaf47c43 (DEFAULT_TABLE table=test-table [id=97fc0548c09047afb8199b025714bb97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:23.169922 22472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1497f5300c47e3866d1ffeaaf47c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:23.169958 22041 catalog_manager.cc:4841] TS 845ed19308ab4670b3078ba5090d5108 (127.20.114.194:43233): tablet 8a2b21dee9524cb9b97d88ca45d10508 (table test-table [id=9f8d448d072c4cf48ed2f93745b42aeb]) successfully deleted
I20260925 22:54:23.170156 22332 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1497f5300c47e3866d1ffeaaf47c43 (DEFAULT_TABLE table=test-table [id=97fc0548c09047afb8199b025714bb97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:23.170380 22332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1497f5300c47e3866d1ffeaaf47c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:23.172065 22586 ts_tablet_manager.cc:1905] T 8a2b21dee9524cb9b97d88ca45d10508 P 462806982da846268dcac68bd7c4f182: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:23.172429 22561 tablet_bootstrap.cc:492] T 5e1497f5300c47e3866d1ffeaaf47c43 P 845ed19308ab4670b3078ba5090d5108: Bootstrap starting.
I20260925 22:54:23.172442 22563 tablet_bootstrap.cc:492] T 5e1497f5300c47e3866d1ffeaaf47c43 P 462806982da846268dcac68bd7c4f182: Bootstrap starting.
I20260925 22:54:23.172951 22192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1497f5300c47e3866d1ffeaaf47c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:23.173353 22561 tablet_bootstrap.cc:654] T 5e1497f5300c47e3866d1ffeaaf47c43 P 845ed19308ab4670b3078ba5090d5108: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:23.175266 22562 tablet_bootstrap.cc:492] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d: Bootstrap starting.
I20260925 22:54:23.175331 22563 tablet_bootstrap.cc:654] T 5e1497f5300c47e3866d1ffeaaf47c43 P 462806982da846268dcac68bd7c4f182: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:23.176155 22586 ts_tablet_manager.cc:1918] T 8a2b21dee9524cb9b97d88ca45d10508 P 462806982da846268dcac68bd7c4f182: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:54:23.176256 22562 tablet_bootstrap.cc:654] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:23.176258 22586 log.cc:1196] T 8a2b21dee9524cb9b97d88ca45d10508 P 462806982da846268dcac68bd7c4f182: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/8a2b21dee9524cb9b97d88ca45d10508
I20260925 22:54:23.176622 22586 ts_tablet_manager.cc:1939] T 8a2b21dee9524cb9b97d88ca45d10508 P 462806982da846268dcac68bd7c4f182: Deleting consensus metadata
I20260925 22:54:23.177047 22039 catalog_manager.cc:4841] TS 1518dd7291014e1b87d449c4ddfecc5d (127.20.114.193:37189): tablet 8a2b21dee9524cb9b97d88ca45d10508 (table test-table [id=9f8d448d072c4cf48ed2f93745b42aeb]) successfully deleted
I20260925 22:54:23.177299 22561 tablet_bootstrap.cc:492] T 5e1497f5300c47e3866d1ffeaaf47c43 P 845ed19308ab4670b3078ba5090d5108: No bootstrap required, opened a new log
I20260925 22:54:23.177382 22561 ts_tablet_manager.cc:1397] T 5e1497f5300c47e3866d1ffeaaf47c43 P 845ed19308ab4670b3078ba5090d5108: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 22:54:23.177793 22561 raft_consensus.cc:348] T 5e1497f5300c47e3866d1ffeaaf47c43 P 845ed19308ab4670b3078ba5090d5108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } }
I20260925 22:54:23.177913 22561 raft_consensus.cc:374] T 5e1497f5300c47e3866d1ffeaaf47c43 P 845ed19308ab4670b3078ba5090d5108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:23.177953 22561 raft_consensus.cc:729] T 5e1497f5300c47e3866d1ffeaaf47c43 P 845ed19308ab4670b3078ba5090d5108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 845ed19308ab4670b3078ba5090d5108, State: Initialized, Role: FOLLOWER
I20260925 22:54:23.178052 22039 catalog_manager.cc:4841] TS 462806982da846268dcac68bd7c4f182 (127.20.114.195:42563): tablet 8a2b21dee9524cb9b97d88ca45d10508 (table test-table [id=9f8d448d072c4cf48ed2f93745b42aeb]) successfully deleted
I20260925 22:54:23.178063 22561 consensus_queue.cc:260] T 5e1497f5300c47e3866d1ffeaaf47c43 P 845ed19308ab4670b3078ba5090d5108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } }
I20260925 22:54:23.178320 22561 ts_tablet_manager.cc:1428] T 5e1497f5300c47e3866d1ffeaaf47c43 P 845ed19308ab4670b3078ba5090d5108: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:23.178781 22563 tablet_bootstrap.cc:492] T 5e1497f5300c47e3866d1ffeaaf47c43 P 462806982da846268dcac68bd7c4f182: No bootstrap required, opened a new log
I20260925 22:54:23.178853 22563 ts_tablet_manager.cc:1397] T 5e1497f5300c47e3866d1ffeaaf47c43 P 462806982da846268dcac68bd7c4f182: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260925 22:54:23.179239 22562 tablet_bootstrap.cc:492] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d: No bootstrap required, opened a new log
I20260925 22:54:23.179343 22562 ts_tablet_manager.cc:1397] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 22:54:23.179679 22563 raft_consensus.cc:348] T 5e1497f5300c47e3866d1ffeaaf47c43 P 462806982da846268dcac68bd7c4f182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } }
I20260925 22:54:23.179697 22562 raft_consensus.cc:348] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } }
I20260925 22:54:23.179785 22563 raft_consensus.cc:374] T 5e1497f5300c47e3866d1ffeaaf47c43 P 462806982da846268dcac68bd7c4f182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:23.179805 22562 raft_consensus.cc:374] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:23.179816 22563 raft_consensus.cc:729] T 5e1497f5300c47e3866d1ffeaaf47c43 P 462806982da846268dcac68bd7c4f182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 462806982da846268dcac68bd7c4f182, State: Initialized, Role: FOLLOWER
I20260925 22:54:23.179847 22562 raft_consensus.cc:729] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1518dd7291014e1b87d449c4ddfecc5d, State: Initialized, Role: FOLLOWER
I20260925 22:54:23.179929 22563 consensus_queue.cc:260] T 5e1497f5300c47e3866d1ffeaaf47c43 P 462806982da846268dcac68bd7c4f182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } }
I20260925 22:54:23.179972 22562 consensus_queue.cc:260] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } }
I20260925 22:54:23.180161 22563 ts_tablet_manager.cc:1428] T 5e1497f5300c47e3866d1ffeaaf47c43 P 462806982da846268dcac68bd7c4f182: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260925 22:54:23.180274 22562 ts_tablet_manager.cc:1428] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:23.441807 22580 raft_consensus.cc:482] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:54:23.441977 22580 raft_consensus.cc:504] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } }
I20260925 22:54:23.442462 22580 leader_election.cc:288] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 845ed19308ab4670b3078ba5090d5108 (127.20.114.194:43233), 462806982da846268dcac68bd7c4f182 (127.20.114.195:42563)
I20260925 22:54:23.442989 22352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1497f5300c47e3866d1ffeaaf47c43" candidate_uuid: "1518dd7291014e1b87d449c4ddfecc5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "845ed19308ab4670b3078ba5090d5108" is_pre_election: true
I20260925 22:54:23.442996 22492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1497f5300c47e3866d1ffeaaf47c43" candidate_uuid: "1518dd7291014e1b87d449c4ddfecc5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "462806982da846268dcac68bd7c4f182" is_pre_election: true
I20260925 22:54:23.443140 22352 raft_consensus.cc:2454] T 5e1497f5300c47e3866d1ffeaaf47c43 P 845ed19308ab4670b3078ba5090d5108 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1518dd7291014e1b87d449c4ddfecc5d in term 0.
I20260925 22:54:23.443159 22492 raft_consensus.cc:2454] T 5e1497f5300c47e3866d1ffeaaf47c43 P 462806982da846268dcac68bd7c4f182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1518dd7291014e1b87d449c4ddfecc5d in term 0.
I20260925 22:54:23.443481 22145 leader_election.cc:302] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1518dd7291014e1b87d449c4ddfecc5d, 845ed19308ab4670b3078ba5090d5108; no voters:
I20260925 22:54:23.443670 22580 raft_consensus.cc:2781] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:54:23.443728 22580 raft_consensus.cc:482] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:54:23.443770 22580 raft_consensus.cc:3037] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:23.444753 22580 raft_consensus.cc:504] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } }
I20260925 22:54:23.445138 22580 leader_election.cc:288] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d [CANDIDATE]: Term 1 election: Requested vote from peers 845ed19308ab4670b3078ba5090d5108 (127.20.114.194:43233), 462806982da846268dcac68bd7c4f182 (127.20.114.195:42563)
I20260925 22:54:23.445447 22352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1497f5300c47e3866d1ffeaaf47c43" candidate_uuid: "1518dd7291014e1b87d449c4ddfecc5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "845ed19308ab4670b3078ba5090d5108"
I20260925 22:54:23.445477 22492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1497f5300c47e3866d1ffeaaf47c43" candidate_uuid: "1518dd7291014e1b87d449c4ddfecc5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "462806982da846268dcac68bd7c4f182"
I20260925 22:54:23.445570 22352 raft_consensus.cc:3037] T 5e1497f5300c47e3866d1ffeaaf47c43 P 845ed19308ab4670b3078ba5090d5108 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:23.445570 22492 raft_consensus.cc:3037] T 5e1497f5300c47e3866d1ffeaaf47c43 P 462806982da846268dcac68bd7c4f182 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:23.446694 22352 raft_consensus.cc:2454] T 5e1497f5300c47e3866d1ffeaaf47c43 P 845ed19308ab4670b3078ba5090d5108 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1518dd7291014e1b87d449c4ddfecc5d in term 1.
I20260925 22:54:23.446694 22492 raft_consensus.cc:2454] T 5e1497f5300c47e3866d1ffeaaf47c43 P 462806982da846268dcac68bd7c4f182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1518dd7291014e1b87d449c4ddfecc5d in term 1.
I20260925 22:54:23.446964 22143 leader_election.cc:302] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1518dd7291014e1b87d449c4ddfecc5d, 462806982da846268dcac68bd7c4f182; no voters:
I20260925 22:54:23.447117 22580 raft_consensus.cc:2781] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:23.447217 22580 raft_consensus.cc:686] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d [term 1 LEADER]: Becoming Leader. State: Replica: 1518dd7291014e1b87d449c4ddfecc5d, State: Running, Role: LEADER
I20260925 22:54:23.447399 22580 consensus_queue.cc:237] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } }
I20260925 22:54:23.449119 22053 catalog_manager.cc:5495] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 1518dd7291014e1b87d449c4ddfecc5d (127.20.114.193). New cstate: current_term: 1 leader_uuid: "1518dd7291014e1b87d449c4ddfecc5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } health_report { overall_health: UNKNOWN } } }
Sep 25 22:54:23 dist-test-slave-x1lh krb5kdc[22010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376861, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:23.490991 22598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:23.482527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:34211 (local address 127.0.0.1:42270)
0925 22:54:23.482940 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:23.482952 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:23.483052 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:23.483276 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:23.483278 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:23.483655 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:23.483748 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:23.483753 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.484484 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.484486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:23.485071 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:23.485079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.485493 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.485496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:23.485555 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:23.486167 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:23.486179 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:23.487637 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:23.489491 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:23.489500 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:23.489506 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:23.489722 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:23.490049 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:23.490052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:23.490053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:23.490128 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:23.490554 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:23.490559 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:23.490663 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 22:54:23.490749 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 22:54:23.490862 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":119,"threads_started":1}
I20260925 22:54:23.491137 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:23.482557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42270 (local address 127.20.114.254:34211)
0925 22:54:23.482821 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:23.482828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:23.483023 (+ 195us) server_negotiation.cc:407] Connection header received
0925 22:54:23.483118 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:23.483120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:23.483159 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:23.483243 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:23.483844 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.484360 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.485166 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.485401 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.485472 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:23.487782 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:23.487785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:23.487787 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:23.487806 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:23.489372 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:23.489856 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:23.489860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:23.489861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:23.489953 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:23.490216 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:23.490219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:23.490220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:23.490455 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:23.490552 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:23.490829 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 22:54:23.490952 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":100,"threads_started":1}
I20260925 22:54:23.494975 22053 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:42270:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 22:54:23.495728 22053 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:54:23.500479 22192 tablet_service.cc:1467] Processing CreateTablet for tablet 89b1a46412004240beb6a33d768e4c89 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4e99b3073e894d0081ef94d6e3d01813]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:54:23.500622 22472 tablet_service.cc:1467] Processing CreateTablet for tablet 89b1a46412004240beb6a33d768e4c89 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4e99b3073e894d0081ef94d6e3d01813]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:54:23.500948 22192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89b1a46412004240beb6a33d768e4c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:23.500948 22472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89b1a46412004240beb6a33d768e4c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:23.503536 22332 tablet_service.cc:1467] Processing CreateTablet for tablet 89b1a46412004240beb6a33d768e4c89 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4e99b3073e894d0081ef94d6e3d01813]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:54:23.503886 22332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89b1a46412004240beb6a33d768e4c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:23.503969 22562 tablet_bootstrap.cc:492] T 89b1a46412004240beb6a33d768e4c89 P 1518dd7291014e1b87d449c4ddfecc5d: Bootstrap starting.
I20260925 22:54:23.503969 22563 tablet_bootstrap.cc:492] T 89b1a46412004240beb6a33d768e4c89 P 462806982da846268dcac68bd7c4f182: Bootstrap starting.
I20260925 22:54:23.504777 22562 tablet_bootstrap.cc:654] T 89b1a46412004240beb6a33d768e4c89 P 1518dd7291014e1b87d449c4ddfecc5d: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:23.504812 22563 tablet_bootstrap.cc:654] T 89b1a46412004240beb6a33d768e4c89 P 462806982da846268dcac68bd7c4f182: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:23.506022 22562 tablet_bootstrap.cc:492] T 89b1a46412004240beb6a33d768e4c89 P 1518dd7291014e1b87d449c4ddfecc5d: No bootstrap required, opened a new log
I20260925 22:54:23.506115 22562 ts_tablet_manager.cc:1397] T 89b1a46412004240beb6a33d768e4c89 P 1518dd7291014e1b87d449c4ddfecc5d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:54:23.506431 22563 tablet_bootstrap.cc:492] T 89b1a46412004240beb6a33d768e4c89 P 462806982da846268dcac68bd7c4f182: No bootstrap required, opened a new log
I20260925 22:54:23.506520 22563 ts_tablet_manager.cc:1397] T 89b1a46412004240beb6a33d768e4c89 P 462806982da846268dcac68bd7c4f182: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:54:23.506538 22562 raft_consensus.cc:348] T 89b1a46412004240beb6a33d768e4c89 P 1518dd7291014e1b87d449c4ddfecc5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } }
I20260925 22:54:23.506673 22562 raft_consensus.cc:374] T 89b1a46412004240beb6a33d768e4c89 P 1518dd7291014e1b87d449c4ddfecc5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:23.506716 22562 raft_consensus.cc:729] T 89b1a46412004240beb6a33d768e4c89 P 1518dd7291014e1b87d449c4ddfecc5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1518dd7291014e1b87d449c4ddfecc5d, State: Initialized, Role: FOLLOWER
I20260925 22:54:23.506842 22562 consensus_queue.cc:260] T 89b1a46412004240beb6a33d768e4c89 P 1518dd7291014e1b87d449c4ddfecc5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } }
I20260925 22:54:23.506958 22563 raft_consensus.cc:348] T 89b1a46412004240beb6a33d768e4c89 P 462806982da846268dcac68bd7c4f182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } }
I20260925 22:54:23.507073 22563 raft_consensus.cc:374] T 89b1a46412004240beb6a33d768e4c89 P 462806982da846268dcac68bd7c4f182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:23.507108 22563 raft_consensus.cc:729] T 89b1a46412004240beb6a33d768e4c89 P 462806982da846268dcac68bd7c4f182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 462806982da846268dcac68bd7c4f182, State: Initialized, Role: FOLLOWER
I20260925 22:54:23.507151 22561 tablet_bootstrap.cc:492] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108: Bootstrap starting.
I20260925 22:54:23.507148 22562 ts_tablet_manager.cc:1428] T 89b1a46412004240beb6a33d768e4c89 P 1518dd7291014e1b87d449c4ddfecc5d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:23.507227 22563 consensus_queue.cc:260] T 89b1a46412004240beb6a33d768e4c89 P 462806982da846268dcac68bd7c4f182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } }
I20260925 22:54:23.507351 22580 tablet_replica.cc:439] T 89b1a46412004240beb6a33d768e4c89 P 1518dd7291014e1b87d449c4ddfecc5d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } } }
I20260925 22:54:23.507468 22563 ts_tablet_manager.cc:1428] T 89b1a46412004240beb6a33d768e4c89 P 462806982da846268dcac68bd7c4f182: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:23.507488 22580 tablet_replica.cc:442] T 89b1a46412004240beb6a33d768e4c89 P 1518dd7291014e1b87d449c4ddfecc5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:54:23.507673 22569 tablet_replica.cc:439] T 89b1a46412004240beb6a33d768e4c89 P 462806982da846268dcac68bd7c4f182: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } } }
I20260925 22:54:23.507794 22569 tablet_replica.cc:442] T 89b1a46412004240beb6a33d768e4c89 P 462806982da846268dcac68bd7c4f182: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:54:23.508065 22561 tablet_bootstrap.cc:654] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:23.509316 22561 tablet_bootstrap.cc:492] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108: No bootstrap required, opened a new log
I20260925 22:54:23.509410 22561 ts_tablet_manager.cc:1397] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:23.509845 22561 raft_consensus.cc:348] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } }
I20260925 22:54:23.509964 22561 raft_consensus.cc:374] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:23.509999 22561 raft_consensus.cc:729] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 845ed19308ab4670b3078ba5090d5108, State: Initialized, Role: FOLLOWER
I20260925 22:54:23.510139 22561 consensus_queue.cc:260] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } }
I20260925 22:54:23.510449 22561 ts_tablet_manager.cc:1428] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:23.510393 22567 tablet_replica.cc:439] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } } }
I20260925 22:54:23.510581 22567 tablet_replica.cc:442] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:54:23.512475 22567 raft_consensus.cc:482] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:54:23.512578 22567 raft_consensus.cc:504] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } }
I20260925 22:54:23.513585 22567 leader_election.cc:288] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1518dd7291014e1b87d449c4ddfecc5d (127.20.114.193:37189), 462806982da846268dcac68bd7c4f182 (127.20.114.195:42563)
I20260925 22:54:23.518157 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:23.514290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:58605 (local address 127.20.114.195:42563)
0925 22:54:23.514403 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:23.514408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:23.514767 (+ 359us) server_negotiation.cc:407] Connection header received
0925 22:54:23.514946 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:23.514949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:23.514993 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:23.515068 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:23.515336 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.515865 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.517168 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.517627 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.517741 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:23.517800 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:23.517967 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 22:54:23.518047 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 22:54:23.518180 22602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:23.514230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:42563 (local address 127.20.114.194:58605)
0925 22:54:23.514688 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:23.514702 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:23.514823 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:23.515090 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:23.515092 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:23.515105 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:23.515228 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:23.515234 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.516012 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.516014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:23.517011 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:23.517020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.517741 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.517744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:23.517840 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:23.517866 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:54:23.517943 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 22:54:23.518010 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":53,"thread_start_us":89,"threads_started":1}
I20260925 22:54:23.518220 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:23.513958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:57287 (local address 127.20.114.193:37189)
0925 22:54:23.514502 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:23.514506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:23.514671 (+ 165us) server_negotiation.cc:407] Connection header received
0925 22:54:23.514778 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:23.514784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:23.514833 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:23.514915 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:23.515208 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.515837 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.517254 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.517646 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.517795 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:23.517844 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:23.518048 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 22:54:23.518115 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470}
I20260925 22:54:23.518610 22492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b1a46412004240beb6a33d768e4c89" candidate_uuid: "845ed19308ab4670b3078ba5090d5108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "462806982da846268dcac68bd7c4f182" is_pre_election: true
I20260925 22:54:23.518647 22603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:23.513860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:37189 (local address 127.20.114.194:57287)
0925 22:54:23.514588 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:23.514603 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:23.514703 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:23.514938 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:23.514941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:23.514957 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:23.515099 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:23.515106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.515944 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.515950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:23.517015 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:23.517025 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.517782 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.517786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:23.517892 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:23.517908 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 22:54:23.518419 (+ 511us) client_negotiation.cc:240] Negotiation successful
0925 22:54:23.518493 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":52,"thread_start_us":110,"threads_started":1}
I20260925 22:54:23.518752 22492 raft_consensus.cc:2454] T 89b1a46412004240beb6a33d768e4c89 P 462806982da846268dcac68bd7c4f182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 845ed19308ab4670b3078ba5090d5108 in term 0.
I20260925 22:54:23.519061 22283 leader_election.cc:302] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 462806982da846268dcac68bd7c4f182, 845ed19308ab4670b3078ba5090d5108; no voters:
I20260925 22:54:23.519088 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b1a46412004240beb6a33d768e4c89" candidate_uuid: "845ed19308ab4670b3078ba5090d5108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1518dd7291014e1b87d449c4ddfecc5d" is_pre_election: true
I20260925 22:54:23.519273 22567 raft_consensus.cc:2781] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:54:23.519325 22212 raft_consensus.cc:2454] T 89b1a46412004240beb6a33d768e4c89 P 1518dd7291014e1b87d449c4ddfecc5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 845ed19308ab4670b3078ba5090d5108 in term 0.
I20260925 22:54:23.519399 22567 raft_consensus.cc:482] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:54:23.519498 22567 raft_consensus.cc:3037] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:23.520363 22567 raft_consensus.cc:504] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } }
I20260925 22:54:23.520666 22567 leader_election.cc:288] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108 [CANDIDATE]: Term 1 election: Requested vote from peers 1518dd7291014e1b87d449c4ddfecc5d (127.20.114.193:37189), 462806982da846268dcac68bd7c4f182 (127.20.114.195:42563)
I20260925 22:54:23.521018 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b1a46412004240beb6a33d768e4c89" candidate_uuid: "845ed19308ab4670b3078ba5090d5108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1518dd7291014e1b87d449c4ddfecc5d"
I20260925 22:54:23.521051 22492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b1a46412004240beb6a33d768e4c89" candidate_uuid: "845ed19308ab4670b3078ba5090d5108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "462806982da846268dcac68bd7c4f182"
I20260925 22:54:23.521128 22492 raft_consensus.cc:3037] T 89b1a46412004240beb6a33d768e4c89 P 462806982da846268dcac68bd7c4f182 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:23.521134 22212 raft_consensus.cc:3037] T 89b1a46412004240beb6a33d768e4c89 P 1518dd7291014e1b87d449c4ddfecc5d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:23.521896 22492 raft_consensus.cc:2454] T 89b1a46412004240beb6a33d768e4c89 P 462806982da846268dcac68bd7c4f182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 845ed19308ab4670b3078ba5090d5108 in term 1.
I20260925 22:54:23.522049 22212 raft_consensus.cc:2454] T 89b1a46412004240beb6a33d768e4c89 P 1518dd7291014e1b87d449c4ddfecc5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 845ed19308ab4670b3078ba5090d5108 in term 1.
I20260925 22:54:23.522183 22283 leader_election.cc:302] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 462806982da846268dcac68bd7c4f182, 845ed19308ab4670b3078ba5090d5108; no voters:
I20260925 22:54:23.522357 22567 raft_consensus.cc:2781] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:23.522570 22567 raft_consensus.cc:686] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108 [term 1 LEADER]: Becoming Leader. State: Replica: 845ed19308ab4670b3078ba5090d5108, State: Running, Role: LEADER
I20260925 22:54:23.522830 22567 consensus_queue.cc:237] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } }
I20260925 22:54:23.524160 22604 tablet_replica.cc:439] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108: TxnStatusTablet state changed. Reason: New leader 845ed19308ab4670b3078ba5090d5108. Latest consensus state: current_term: 1 leader_uuid: "845ed19308ab4670b3078ba5090d5108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } } }
I20260925 22:54:23.524289 22604 tablet_replica.cc:442] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:54:23.524567 22606 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 22:54:23.524718 22606 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 22:54:23.525002 22053 catalog_manager.cc:5495] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108 reported cstate change: term changed from 0 to 1, leader changed from <none> to 845ed19308ab4670b3078ba5090d5108 (127.20.114.194). New cstate: current_term: 1 leader_uuid: "845ed19308ab4670b3078ba5090d5108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } health_report { overall_health: UNKNOWN } } }
I20260925 22:54:23.537909 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:23.533853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48904 (local address 127.20.114.194:43233)
0925 22:54:23.533977 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:23.533981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:23.534072 (+ 91us) server_negotiation.cc:407] Connection header received
0925 22:54:23.534229 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:23.534233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:23.534297 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:23.534410 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:23.534627 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.535382 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.536193 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.536528 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.536632 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:23.537032 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:23.537541 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:23.537676 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:23.537722 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 22:54:23.537782 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:54:23.537883 22598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:23.533750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:43233 (local address 127.0.0.1:48904)
0925 22:54:23.533993 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:23.534011 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:23.534108 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:23.534439 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:23.534441 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:23.534447 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:54:23.534531 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:23.534535 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.535494 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.535496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:23.536084 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:23.536091 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.536725 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.536728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:23.536797 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:23.536840 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:54:23.537650 (+ 810us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:54:23.537661 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 22:54:23.537703 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 22:54:23.537781 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260925 22:54:23.540602 22492 raft_consensus.cc:1261] T 89b1a46412004240beb6a33d768e4c89 P 462806982da846268dcac68bd7c4f182 [term 1 FOLLOWER]: Refusing update from remote peer 845ed19308ab4670b3078ba5090d5108: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:23.541069 22212 raft_consensus.cc:1261] T 89b1a46412004240beb6a33d768e4c89 P 1518dd7291014e1b87d449c4ddfecc5d [term 1 FOLLOWER]: Refusing update from remote peer 845ed19308ab4670b3078ba5090d5108: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:23.541070 22567 consensus_queue.cc:1035] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:23.541656 22567 consensus_queue.cc:1035] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:23.542441 22569 tablet_replica.cc:439] T 89b1a46412004240beb6a33d768e4c89 P 462806982da846268dcac68bd7c4f182: TxnStatusTablet state changed. Reason: New leader 845ed19308ab4670b3078ba5090d5108. Latest consensus state: current_term: 1 leader_uuid: "845ed19308ab4670b3078ba5090d5108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } } }
I20260925 22:54:23.542542 22569 tablet_replica.cc:442] T 89b1a46412004240beb6a33d768e4c89 P 462806982da846268dcac68bd7c4f182: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:54:23.543733 22580 tablet_replica.cc:439] T 89b1a46412004240beb6a33d768e4c89 P 1518dd7291014e1b87d449c4ddfecc5d: TxnStatusTablet state changed. Reason: New leader 845ed19308ab4670b3078ba5090d5108. Latest consensus state: current_term: 1 leader_uuid: "845ed19308ab4670b3078ba5090d5108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } } }
I20260925 22:54:23.543848 22580 tablet_replica.cc:442] T 89b1a46412004240beb6a33d768e4c89 P 1518dd7291014e1b87d449c4ddfecc5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:54:23.545051 22604 tablet_replica.cc:439] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "845ed19308ab4670b3078ba5090d5108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } } }
I20260925 22:54:23.545100 22577 mvcc.cc:205] Tried to move back new op lower bound from 7333383633057480704 to 7333383632992030720. Current Snapshot: MvccSnapshot[applied={T|T < 7333383633057480704}]
I20260925 22:54:23.545149 22604 tablet_replica.cc:442] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:54:23.545593 22569 tablet_replica.cc:439] T 89b1a46412004240beb6a33d768e4c89 P 462806982da846268dcac68bd7c4f182: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "845ed19308ab4670b3078ba5090d5108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } } }
I20260925 22:54:23.545724 22569 tablet_replica.cc:442] T 89b1a46412004240beb6a33d768e4c89 P 462806982da846268dcac68bd7c4f182: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:54:23.546234 22576 mvcc.cc:205] Tried to move back new op lower bound from 7333383633057480704 to 7333383632992030720. Current Snapshot: MvccSnapshot[applied={T|T < 7333383633057480704 or (T in {7333383633057480704})}]
I20260925 22:54:23.548743 22567 tablet_replica.cc:439] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "845ed19308ab4670b3078ba5090d5108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } } }
I20260925 22:54:23.548846 22567 tablet_replica.cc:442] T 89b1a46412004240beb6a33d768e4c89 P 845ed19308ab4670b3078ba5090d5108: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:54:23.550096 22580 tablet_replica.cc:439] T 89b1a46412004240beb6a33d768e4c89 P 1518dd7291014e1b87d449c4ddfecc5d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "845ed19308ab4670b3078ba5090d5108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1518dd7291014e1b87d449c4ddfecc5d" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37189 } } peers { permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 } } peers { permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 } } }
I20260925 22:54:23.550207 22580 tablet_replica.cc:442] T 89b1a46412004240beb6a33d768e4c89 P 1518dd7291014e1b87d449c4ddfecc5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:54:23.550298 22575 mvcc.cc:205] Tried to move back new op lower bound from 7333383633057480704 to 7333383632992030720. Current Snapshot: MvccSnapshot[applied={T|T < 7333383633057480704 or (T in {7333383633057480704})}]
I20260925 22:54:23.559286 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:23.555764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:37713 (local address 127.20.114.194:43233)
0925 22:54:23.555888 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:23.555893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:23.556149 (+ 256us) server_negotiation.cc:407] Connection header received
0925 22:54:23.556273 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:23.556275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:23.556316 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:23.556378 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:23.556623 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.557169 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.558346 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.558554 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.558628 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:23.558925 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:23.558988 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:23.559036 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:23.559151 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 22:54:23.559187 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:54:23.559381 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:23.555638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:43233 (local address 127.20.114.193:37713)
0925 22:54:23.556054 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:23.556069 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:23.556171 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:23.556409 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:23.556412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:23.556420 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:54:23.556527 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:23.556533 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.557265 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.557268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:23.558239 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:23.558249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.558653 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.558657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:23.558744 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:23.558796 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:54:23.559062 (+ 266us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:54:23.559069 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 22:54:23.559135 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 22:54:23.559218 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":116,"threads_started":1}
I20260925 22:54:23.565573 22352 raft_consensus.cc:1261] T 5e1497f5300c47e3866d1ffeaaf47c43 P 845ed19308ab4670b3078ba5090d5108 [term 1 FOLLOWER]: Refusing update from remote peer 1518dd7291014e1b87d449c4ddfecc5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:23.565568 22492 raft_consensus.cc:1261] T 5e1497f5300c47e3866d1ffeaaf47c43 P 462806982da846268dcac68bd7c4f182 [term 1 FOLLOWER]: Refusing update from remote peer 1518dd7291014e1b87d449c4ddfecc5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:23.565961 22572 consensus_queue.cc:1035] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "462806982da846268dcac68bd7c4f182" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:23.566167 22580 consensus_queue.cc:1035] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "845ed19308ab4670b3078ba5090d5108" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:23.587523 22619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:23.583561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:37189 (local address 127.20.114.194:41205)
0925 22:54:23.584221 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:23.584234 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:23.584318 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:23.584576 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:23.584578 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:23.584583 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:54:23.584679 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:23.584683 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.585506 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.585512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:23.586333 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:23.586343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.586796 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.586800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:23.586887 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:23.586943 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:54:23.587235 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:54:23.587242 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 22:54:23.587301 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 22:54:23.587373 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":92,"threads_started":1}
I20260925 22:54:23.587504 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:23.583870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:41205 (local address 127.20.114.193:37189)
0925 22:54:23.584135 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:23.584140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:23.584302 (+ 162us) server_negotiation.cc:407] Connection header received
0925 22:54:23.584437 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:23.584441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:23.584490 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:23.584574 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:23.584791 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.585390 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.586453 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.586692 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.586799 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:23.587071 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:23.587147 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:23.587221 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:23.587310 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 22:54:23.587359 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260925 22:54:23.610522 22050 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:23.610684 22050 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:23.612720 22050 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b40dd73e9ea4dd1828001acb5e9ff83: Sending DeleteTablet for 3 replicas of tablet 5e1497f5300c47e3866d1ffeaaf47c43
I20260925 22:54:23.613261 22192 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e1497f5300c47e3866d1ffeaaf47c43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:23 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:42284
I20260925 22:54:23.613384 22332 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e1497f5300c47e3866d1ffeaaf47c43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:23 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:48890
I20260925 22:54:23.613441 22587 tablet_replica.cc:330] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d: stopping tablet replica
I20260925 22:54:23.613456 22472 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e1497f5300c47e3866d1ffeaaf47c43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:23 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:35882
I20260925 22:54:23.613562 22585 tablet_replica.cc:330] T 5e1497f5300c47e3866d1ffeaaf47c43 P 845ed19308ab4670b3078ba5090d5108: stopping tablet replica
I20260925 22:54:23.613555 22587 raft_consensus.cc:2229] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:54:23.613607 22586 tablet_replica.cc:330] T 5e1497f5300c47e3866d1ffeaaf47c43 P 462806982da846268dcac68bd7c4f182: stopping tablet replica
I20260925 22:54:23.613651 22585 raft_consensus.cc:2229] T 5e1497f5300c47e3866d1ffeaaf47c43 P 845ed19308ab4670b3078ba5090d5108 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:23.613698 22586 raft_consensus.cc:2229] T 5e1497f5300c47e3866d1ffeaaf47c43 P 462806982da846268dcac68bd7c4f182 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:23.613735 22587 raft_consensus.cc:2258] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:23.613750 22585 raft_consensus.cc:2258] T 5e1497f5300c47e3866d1ffeaaf47c43 P 845ed19308ab4670b3078ba5090d5108 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:23.613777 22586 raft_consensus.cc:2258] T 5e1497f5300c47e3866d1ffeaaf47c43 P 462806982da846268dcac68bd7c4f182 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:23.614527 22586 ts_tablet_manager.cc:1905] T 5e1497f5300c47e3866d1ffeaaf47c43 P 462806982da846268dcac68bd7c4f182: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:23.614639 22585 ts_tablet_manager.cc:1905] T 5e1497f5300c47e3866d1ffeaaf47c43 P 845ed19308ab4670b3078ba5090d5108: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:23.617357 22586 ts_tablet_manager.cc:1918] T 5e1497f5300c47e3866d1ffeaaf47c43 P 462806982da846268dcac68bd7c4f182: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:54:23.617432 22586 log.cc:1196] T 5e1497f5300c47e3866d1ffeaaf47c43 P 462806982da846268dcac68bd7c4f182: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/5e1497f5300c47e3866d1ffeaaf47c43
I20260925 22:54:23.617758 22585 ts_tablet_manager.cc:1918] T 5e1497f5300c47e3866d1ffeaaf47c43 P 845ed19308ab4670b3078ba5090d5108: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:54:23.617832 22585 log.cc:1196] T 5e1497f5300c47e3866d1ffeaaf47c43 P 845ed19308ab4670b3078ba5090d5108: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/5e1497f5300c47e3866d1ffeaaf47c43
I20260925 22:54:23.617832 22586 ts_tablet_manager.cc:1939] T 5e1497f5300c47e3866d1ffeaaf47c43 P 462806982da846268dcac68bd7c4f182: Deleting consensus metadata
I20260925 22:54:23.618083 22585 ts_tablet_manager.cc:1939] T 5e1497f5300c47e3866d1ffeaaf47c43 P 845ed19308ab4670b3078ba5090d5108: Deleting consensus metadata
I20260925 22:54:23.618165 22587 ts_tablet_manager.cc:1905] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:23.618479 22039 catalog_manager.cc:4841] TS 462806982da846268dcac68bd7c4f182 (127.20.114.195:42563): tablet 5e1497f5300c47e3866d1ffeaaf47c43 (table test-table [id=97fc0548c09047afb8199b025714bb97]) successfully deleted
I20260925 22:54:23.618685 22041 catalog_manager.cc:4841] TS 845ed19308ab4670b3078ba5090d5108 (127.20.114.194:43233): tablet 5e1497f5300c47e3866d1ffeaaf47c43 (table test-table [id=97fc0548c09047afb8199b025714bb97]) successfully deleted
I20260925 22:54:23.620323 22587 ts_tablet_manager.cc:1918] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:54:23.620373 22587 log.cc:1196] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376856092611-20939-0/minicluster-data/ts-0/wal/wals/5e1497f5300c47e3866d1ffeaaf47c43
I20260925 22:54:23.620553 22587 ts_tablet_manager.cc:1939] T 5e1497f5300c47e3866d1ffeaaf47c43 P 1518dd7291014e1b87d449c4ddfecc5d: Deleting consensus metadata
I20260925 22:54:23.621347 22039 catalog_manager.cc:4841] TS 1518dd7291014e1b87d449c4ddfecc5d (127.20.114.193:37189): tablet 5e1497f5300c47e3866d1ffeaaf47c43 (table test-table [id=97fc0548c09047afb8199b025714bb97]) successfully deleted
Sep 25 22:54:23 dist-test-slave-x1lh krb5kdc[22010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376862, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.114.193@KRBTEST.COM
I20260925 22:54:23.630542 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:23.617217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42306 (local address 127.20.114.193:37189)
0925 22:54:23.617337 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:23.617342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:23.617356 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:54:23.617397 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:23.617400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:23.617450 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:23.617532 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:23.621656 (+ 4124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.622436 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.623224 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.623568 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.623668 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:23.626929 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:23.626941 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:23.626959 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:23.626985 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:23.628838 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:23.629274 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:23.629281 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:23.629283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:23.629395 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:23.629633 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:23.629636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:23.629638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:23.629927 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:23.630133 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:23.630260 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 22:54:23.630385 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 22:54:23.630996 22249 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42306
I20260925 22:54:23.640483 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:23.632690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42286 (local address 127.20.114.254:34211)
0925 22:54:23.632820 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:23.632827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:23.632905 (+ 78us) server_negotiation.cc:407] Connection header received
0925 22:54:23.633091 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:23.633094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:23.633146 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:23.633229 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:23.633860 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.634682 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.635466 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:23.635824 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:23.635922 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:23.637243 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:23.637247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:23.637250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:23.637277 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:23.638824 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:23.639192 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:23.639196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:23.639198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:23.639289 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:23.639634 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:23.639637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:23.639639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:23.639861 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:23.639993 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:23.640182 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 22:54:23.640310 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 22:54:23.640935 22050 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42286
I20260925 22:54:23.642023 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 22127
I20260925 22:54:23.650576 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 22267
I20260925 22:54:23.656816 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 22407
I20260925 22:54:23.662174 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 22023
2026-09-25T22:54:23Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3009 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:54:23 dist-test-slave-x1lh krb5kdc[22627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:54:23 dist-test-slave-x1lh krb5kdc[22627](info): set up 2 sockets
Sep 25 22:54:23 dist-test-slave-x1lh krb5kdc[22627](info): commencing operation
krb5kdc: starting...
W20260925 22:54:24.696717 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:54:24 dist-test-slave-x1lh krb5kdc[22627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:54:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:54:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:54:24.816417 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:42893
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:39373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:42893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:24.925438 22640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:24.925683 22640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:24.925720 22640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:24.929590 22640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:24.929636 22640 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 22:54:24.929652 22640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:24.929670 22640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:24.929685 22640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:24.929700 22640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:24.935470 22640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:39373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:42893
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:42893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:24.936415 22640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:24.937405 22640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:24.945609 22646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:24.945605 22648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:24.945652 22640 server_base.cc:1034] running on GCE node
W20260925 22:54:24.945605 22645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:24.946382 22640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:24.947510 22640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:24.948671 22640 hybrid_clock.cc:648] HybridClock initialized: now 1790376864948660 us; error 25 us; skew 500 ppm
Sep 25 22:54:24 dist-test-slave-x1lh krb5kdc[22627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376864, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:24.951402 22640 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:54:24.952288 22640 webserver.cc:460] Webserver started at http://127.20.114.254:46737/ using document root <none> and password file <none>
I20260925 22:54:24.952821 22640 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:24.952936 22640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:24.953167 22640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:24.955112 22640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "a9351c311fc446649b41e5bedd96812d"
format_stamp: "Formatted at 2026-09-25 22:54:24 on dist-test-slave-x1lh"
server_key: "829a8e6a2f08113543380b7084e1fa8e"
server_key_iv: "9cfd7abcf0d1698b5493004550bd04c8"
server_key_version: "encryptionkey@0"
I20260925 22:54:24.955657 22640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "a9351c311fc446649b41e5bedd96812d"
format_stamp: "Formatted at 2026-09-25 22:54:24 on dist-test-slave-x1lh"
server_key: "829a8e6a2f08113543380b7084e1fa8e"
server_key_iv: "9cfd7abcf0d1698b5493004550bd04c8"
server_key_version: "encryptionkey@0"
I20260925 22:54:24.959259 22640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 22:54:24.961611 22655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:24.962976 22640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:54:24.963114 22640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "a9351c311fc446649b41e5bedd96812d"
format_stamp: "Formatted at 2026-09-25 22:54:24 on dist-test-slave-x1lh"
server_key: "829a8e6a2f08113543380b7084e1fa8e"
server_key_iv: "9cfd7abcf0d1698b5493004550bd04c8"
server_key_version: "encryptionkey@0"
I20260925 22:54:24.963222 22640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:24.974496 22640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:24.977061 22640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:24.977248 22640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:24.985383 22640 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:42893
I20260925 22:54:24.985411 22717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:42893 every 8 connection(s)
I20260925 22:54:24.986270 22640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:54:24.989012 22718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:24.992522 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 22640
I20260925 22:54:24.992610 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:54:24.992923 20939 external_mini_cluster.cc:1428] Setting key a8b0a44005223b1f6912215aaecbd0a4
I20260925 22:54:24.995046 22718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d: Bootstrap starting.
I20260925 22:54:24.997797 22718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:24.998638 22718 log.cc:824] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:25.000782 22718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d: No bootstrap required, opened a new log
Sep 25 22:54:24 dist-test-slave-x1lh krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:25.005000 22718 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9351c311fc446649b41e5bedd96812d" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 42893 } }
I20260925 22:54:25.005314 22718 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:25.005390 22718 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9351c311fc446649b41e5bedd96812d, State: Initialized, Role: FOLLOWER
I20260925 22:54:25.005883 22718 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9351c311fc446649b41e5bedd96812d" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 42893 } }
I20260925 22:54:25.006002 22718 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:54:25.006081 22718 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:54:25.006181 22718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:25.007099 22718 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9351c311fc446649b41e5bedd96812d" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 42893 } }
I20260925 22:54:25.007457 22718 leader_election.cc:302] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9351c311fc446649b41e5bedd96812d; no voters:
I20260925 22:54:25.007798 22718 leader_election.cc:288] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:54:25.007882 22723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:25.008059 22723 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d [term 1 LEADER]: Becoming Leader. State: Replica: a9351c311fc446649b41e5bedd96812d, State: Running, Role: LEADER
I20260925 22:54:25.008246 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:24.994395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36676 (local address 127.20.114.254:42893)
0925 22:54:24.994808 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:24.994827 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:24.994868 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:54:24.995657 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:24.995680 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:24.996100 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:24.996485 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:24.997173 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:24.998679 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:24.999711 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.000369 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.000521 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.003148 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:25.003153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:25.003165 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:25.003189 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:25.004848 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:25.005400 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:25.005408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:25.005417 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:25.005587 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:25.005895 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:25.005899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:25.005901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:25.006457 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:25.006697 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:25.006933 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 22:54:25.007269 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":96,"threads_started":1}
I20260925 22:54:25.008356 22723 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9351c311fc446649b41e5bedd96812d" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 42893 } }
I20260925 22:54:25.009093 22718 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:25.010757 22725 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9351c311fc446649b41e5bedd96812d. Latest consensus state: current_term: 1 leader_uuid: "a9351c311fc446649b41e5bedd96812d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9351c311fc446649b41e5bedd96812d" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 42893 } } }
I20260925 22:54:25.011029 22724 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9351c311fc446649b41e5bedd96812d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9351c311fc446649b41e5bedd96812d" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 42893 } } }
I20260925 22:54:25.011145 22724 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:25.011390 22725 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d [sys.catalog]: This master's current role is: LEADER
W20260925 22:54:25.013880 22737 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:54:25.013969 22737 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:54:25.014050 22738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:54:25.014710 22738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:54:25.020228 22738 catalog_manager.cc:1347] Generated new cluster ID: 7c7cd2298e8948a9865127820e7953b9
I20260925 22:54:25.020295 22738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:25.036523 22738 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:54:25.037865 22738 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:25.046058 22738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d: Generated new TSK 0
I20260925 22:54:25.046851 22738 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.114.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
I20260925 22:54:25.078255 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:42893
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:39373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:25.189083 22744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:25.189388 22744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:25.189473 22744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:25.193617 22744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:25.193672 22744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:25.193711 22744 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 22:54:25.193759 22744 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:54:25.193817 22744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:54:25.198963 22744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:39373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.114.254:42893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:25.199973 22744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:25.201022 22744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:25.210372 22752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:25.210398 22749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:25.210547 22744 server_base.cc:1034] running on GCE node
W20260925 22:54:25.210332 22750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:25.211048 22744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:25.211730 22744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:25.212952 22744 hybrid_clock.cc:648] HybridClock initialized: now 1790376865212909 us; error 47 us; skew 500 ppm
Sep 25 22:54:25 dist-test-slave-x1lh krb5kdc[22627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376865, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:25.215715 22744 init.cc:377] Logged in from keytab as kudu/127.20.114.193@KRBTEST.COM (short username kudu)
I20260925 22:54:25.216635 22744 webserver.cc:460] Webserver started at http://127.20.114.193:44667/ using document root <none> and password file <none>
I20260925 22:54:25.217216 22744 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:25.217274 22744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:25.217435 22744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:25.219405 22744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "a3525409ceea4c7da73004ad65738ffc"
format_stamp: "Formatted at 2026-09-25 22:54:25 on dist-test-slave-x1lh"
server_key: "a0990edb1d4d9456f60423228552a2b8"
server_key_iv: "478c21e4894e8fa099795ae350eea839"
server_key_version: "encryptionkey@0"
I20260925 22:54:25.219906 22744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "a3525409ceea4c7da73004ad65738ffc"
format_stamp: "Formatted at 2026-09-25 22:54:25 on dist-test-slave-x1lh"
server_key: "a0990edb1d4d9456f60423228552a2b8"
server_key_iv: "478c21e4894e8fa099795ae350eea839"
server_key_version: "encryptionkey@0"
I20260925 22:54:25.223462 22744 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260925 22:54:25.225821 22759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:25.227023 22744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:25.227152 22744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "a3525409ceea4c7da73004ad65738ffc"
format_stamp: "Formatted at 2026-09-25 22:54:25 on dist-test-slave-x1lh"
server_key: "a0990edb1d4d9456f60423228552a2b8"
server_key_iv: "478c21e4894e8fa099795ae350eea839"
server_key_version: "encryptionkey@0"
I20260925 22:54:25.227255 22744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:25.246229 22744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:25.249156 22744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:25.249375 22744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:25.250859 22744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:25.250911 22744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:25.250950 22744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:25.251006 22744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:54:25 dist-test-slave-x1lh krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376865, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:25.262105 22744 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:44777
I20260925 22:54:25.262465 22874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:44777 every 8 connection(s)
I20260925 22:54:25.263132 22744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:54:25.264333 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 22744
I20260925 22:54:25.264459 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:54:25.264756 20939 external_mini_cluster.cc:1428] Setting key 8ab324f13767be7cdc2e0908af788892
I20260925 22:54:25.265334 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:25.251587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:57147 (local address 127.20.114.254:42893)
0925 22:54:25.251708 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:25.251712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:25.252820 (+ 1108us) server_negotiation.cc:407] Connection header received
0925 22:54:25.253871 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:25.253876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:25.253937 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:25.253999 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:25.255934 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.256529 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.257727 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.257978 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.258094 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.261140 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:25.261143 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:25.261145 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:25.261168 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:25.262894 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:25.263486 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:25.263490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:25.263493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:25.263585 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:25.263979 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:25.263982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:25.263984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:25.264242 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:25.264331 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:25.265008 (+ 677us) server_negotiation.cc:299] Negotiation successful
0925 22:54:25.265161 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:54:25.266148 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:25.252159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:42893 (local address 127.20.114.193:57147)
0925 22:54:25.252657 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:25.252698 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:25.253584 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:25.254502 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:25.254531 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:25.255040 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:25.255767 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:25.255789 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.256653 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.256657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:25.257602 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:25.257610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.258144 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.258148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:25.258316 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.259002 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:25.259021 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:25.260938 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:25.263001 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:25.263006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:25.263020 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:25.263353 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:25.263692 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:25.263695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:25.263697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:25.263846 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:25.264352 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:25.264360 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:25.264660 (+ 300us) client_negotiation.cc:769] Sending connection context
0925 22:54:25.264856 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 22:54:25.265208 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":141,"threads_started":1}
I20260925 22:54:25.267529 22875 heartbeater.cc:347] Connected to a master server at 127.20.114.254:42893
I20260925 22:54:25.267668 22875 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:25.268227 22875 heartbeater.cc:511] Master 127.20.114.254:42893 requested a full tablet report, sending...
I20260925 22:54:25.269902 22671 ts_manager.cc:194] Registered new tserver with Master: a3525409ceea4c7da73004ad65738ffc (127.20.114.193:44777)
I20260925 22:54:25.271034 22671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:57147
WARNING: no policy specified for kudu/127.20.114.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.194@KRBTEST.COM" created.
I20260925 22:54:25.279959 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:25.269984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:42893 (local address 127.20.114.193:50381)
0925 22:54:25.273063 (+ 3079us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:25.273076 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:25.273193 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:25.273439 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:25.273442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:25.273646 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:25.273735 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:25.273740 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.274477 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.274480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:25.275108 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:25.275116 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.275561 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.275565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:25.275655 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.276211 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:25.276223 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:25.276534 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:25.278383 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:25.278386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:25.278388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:25.278578 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:25.278940 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:25.278943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:25.278945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:25.279040 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:25.279518 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:25.279520 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:25.279597 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 22:54:25.279685 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 22:54:25.279814 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3006,"thread_start_us":82,"threads_started":1}
I20260925 22:54:25.280086 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:25.270051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:50381 (local address 127.20.114.254:42893)
0925 22:54:25.270152 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:25.270156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:25.273148 (+ 2992us) server_negotiation.cc:407] Connection header received
0925 22:54:25.273293 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:25.273297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:25.273336 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:25.273425 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:25.273849 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.274358 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.275217 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.275453 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.275576 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.276667 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:25.276670 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:25.276673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:25.276699 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:25.278252 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:25.278674 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:25.278679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:25.278680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:25.278771 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:25.279142 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:25.279146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:25.279150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:25.279408 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:25.279519 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:25.279769 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 22:54:25.279906 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
I20260925 22:54:25.293581 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:42893
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:39373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:25.410138 22884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:25.410388 22884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:25.410444 22884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:25.414482 22884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:25.414539 22884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:25.414580 22884 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 22:54:25.414594 22884 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:54:25.414633 22884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:54:25.419835 22884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:39373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.114.254:42893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:25.420799 22884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:25.421823 22884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:25.431209 22892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:25.431149 22890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:25.431142 22889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:25.431551 22884 server_base.cc:1034] running on GCE node
I20260925 22:54:25.431900 22884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:25.432488 22884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:25.433666 22884 hybrid_clock.cc:648] HybridClock initialized: now 1790376865433641 us; error 34 us; skew 500 ppm
Sep 25 22:54:25 dist-test-slave-x1lh krb5kdc[22627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376865, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:25.436237 22884 init.cc:377] Logged in from keytab as kudu/127.20.114.194@KRBTEST.COM (short username kudu)
I20260925 22:54:25.437209 22884 webserver.cc:460] Webserver started at http://127.20.114.194:36167/ using document root <none> and password file <none>
I20260925 22:54:25.437748 22884 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:25.437806 22884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:25.438010 22884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:25.439929 22884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "036521f784e04ecca2343b74e9fda6bf"
format_stamp: "Formatted at 2026-09-25 22:54:25 on dist-test-slave-x1lh"
server_key: "82ad5d132583579a822740fe8f9fa324"
server_key_iv: "c63de84dcdebfcd30445e4613081e153"
server_key_version: "encryptionkey@0"
I20260925 22:54:25.440474 22884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "036521f784e04ecca2343b74e9fda6bf"
format_stamp: "Formatted at 2026-09-25 22:54:25 on dist-test-slave-x1lh"
server_key: "82ad5d132583579a822740fe8f9fa324"
server_key_iv: "c63de84dcdebfcd30445e4613081e153"
server_key_version: "encryptionkey@0"
I20260925 22:54:25.443975 22884 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:54:25.446271 22899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:25.447538 22884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:54:25.447679 22884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "036521f784e04ecca2343b74e9fda6bf"
format_stamp: "Formatted at 2026-09-25 22:54:25 on dist-test-slave-x1lh"
server_key: "82ad5d132583579a822740fe8f9fa324"
server_key_iv: "c63de84dcdebfcd30445e4613081e153"
server_key_version: "encryptionkey@0"
I20260925 22:54:25.447789 22884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:25.470811 22884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:25.473822 22884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:25.474021 22884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:25.475409 22884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:25.475481 22884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:25.475548 22884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:25.475595 22884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:54:25 dist-test-slave-x1lh krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376865, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:25.486209 22884 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:39451
I20260925 22:54:25.486726 23014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:39451 every 8 connection(s)
I20260925 22:54:25.487179 22884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:54:25.489641 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 22884
I20260925 22:54:25.489686 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:25.476262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:37981 (local address 127.20.114.254:42893)
0925 22:54:25.476393 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:25.476397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:25.477485 (+ 1088us) server_negotiation.cc:407] Connection header received
0925 22:54:25.478696 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:25.478699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:25.478753 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:25.478876 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:25.480394 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.481293 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.482702 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.482969 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.483070 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.485781 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:25.485785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:25.485788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:25.485821 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:25.487506 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:25.488110 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:25.488113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:25.488115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:25.488210 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:25.488568 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:25.488573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:25.488575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:25.488768 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:25.488875 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:25.489399 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 22:54:25.489525 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:54:25.489784 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:54:25.490088 20939 external_mini_cluster.cc:1428] Setting key a88777390fa97db0a80d6ad4a5b5890e
I20260925 22:54:25.490589 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:25.476793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:42893 (local address 127.20.114.194:37981)
0925 22:54:25.477318 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:25.477372 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:25.478414 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:25.479091 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:25.479100 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:25.479546 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:25.480255 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:25.480276 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.481439 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.481443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:25.482575 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:25.482585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.483106 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.483110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:25.483288 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.483877 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:25.483893 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:25.485593 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:25.487624 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:25.487630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:25.487644 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:25.487979 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:25.488323 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:25.488326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:25.488328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:25.488463 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:25.488908 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:25.488915 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:25.489145 (+ 230us) client_negotiation.cc:769] Sending connection context
0925 22:54:25.489344 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 22:54:25.489707 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":125,"threads_started":1}
I20260925 22:54:25.491966 23015 heartbeater.cc:347] Connected to a master server at 127.20.114.254:42893
I20260925 22:54:25.492123 23015 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:25.492708 23015 heartbeater.cc:511] Master 127.20.114.254:42893 requested a full tablet report, sending...
I20260925 22:54:25.493790 22672 ts_manager.cc:194] Registered new tserver with Master: 036521f784e04ecca2343b74e9fda6bf (127.20.114.194:39451)
I20260925 22:54:25.494580 22672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:37981
WARNING: no policy specified for kudu/127.20.114.195@KRBTEST.COM; defaulting to no policy
I20260925 22:54:25.501823 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:25.494133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:42893 (local address 127.20.114.194:40347)
0925 22:54:25.494435 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:25.494455 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:25.494556 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:25.494799 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:25.494805 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:25.495029 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:25.495181 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:25.495191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.496272 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.496279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:25.497116 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:25.497125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.497577 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.497580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:25.497672 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.498202 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:25.498215 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:25.498528 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:25.500320 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:25.500324 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:25.500326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:25.500524 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:25.500910 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:25.500913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:25.500915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:25.501001 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:25.501381 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:25.501384 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:25.501455 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 22:54:25.501536 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 22:54:25.501636 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":98,"threads_started":1}
I20260925 22:54:25.501832 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:25.494415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:40347 (local address 127.20.114.254:42893)
0925 22:54:25.494513 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:25.494517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:25.494531 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:54:25.494658 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:25.494662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:25.494699 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:25.494777 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:25.495305 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.496128 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.497240 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.497468 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.497577 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.498659 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:25.498663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:25.498666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:25.498688 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:25.500213 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:25.500622 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:25.500626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:25.500628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:25.500737 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:25.501084 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:25.501087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:25.501088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:25.501283 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:25.501364 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:25.501572 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 22:54:25.501656 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
I20260925 22:54:25.517896 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:42893
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:39373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:25.626101 23024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:25.626368 23024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:25.626423 23024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:25.630793 23024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:25.630849 23024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:25.630910 23024 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 22:54:25.630942 23024 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:54:25.631006 23024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:54:25.636382 23024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:39373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.114.254:42893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:25.637493 23024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:25.638554 23024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:25.648118 23029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:25.648124 23030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:25.648128 23032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:25.648339 23024 server_base.cc:1034] running on GCE node
I20260925 22:54:25.648756 23024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:25.649439 23024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:25.650626 23024 hybrid_clock.cc:648] HybridClock initialized: now 1790376865650599 us; error 48 us; skew 500 ppm
Sep 25 22:54:25 dist-test-slave-x1lh krb5kdc[22627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376865, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:25.653375 23024 init.cc:377] Logged in from keytab as kudu/127.20.114.195@KRBTEST.COM (short username kudu)
I20260925 22:54:25.654249 23024 webserver.cc:460] Webserver started at http://127.20.114.195:45823/ using document root <none> and password file <none>
I20260925 22:54:25.654801 23024 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:25.654897 23024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:25.655120 23024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:25.657130 23024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "3ffde459ed8a47379bfe0a0b314656a0"
format_stamp: "Formatted at 2026-09-25 22:54:25 on dist-test-slave-x1lh"
server_key: "c7577d1f067a6770a89ad9c9f6eed9ce"
server_key_iv: "837aac934012398661bee89d08eaf180"
server_key_version: "encryptionkey@0"
I20260925 22:54:25.657671 23024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "3ffde459ed8a47379bfe0a0b314656a0"
format_stamp: "Formatted at 2026-09-25 22:54:25 on dist-test-slave-x1lh"
server_key: "c7577d1f067a6770a89ad9c9f6eed9ce"
server_key_iv: "837aac934012398661bee89d08eaf180"
server_key_version: "encryptionkey@0"
I20260925 22:54:25.661262 23024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:54:25.663602 23039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:25.664937 23024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:25.665056 23024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "3ffde459ed8a47379bfe0a0b314656a0"
format_stamp: "Formatted at 2026-09-25 22:54:25 on dist-test-slave-x1lh"
server_key: "c7577d1f067a6770a89ad9c9f6eed9ce"
server_key_iv: "837aac934012398661bee89d08eaf180"
server_key_version: "encryptionkey@0"
I20260925 22:54:25.665162 23024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:25.674918 23024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:25.678066 23024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:25.678272 23024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:25.679703 23024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:25.679780 23024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:25.679840 23024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:25.679895 23024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:25.690272 23024 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:43441
I20260925 22:54:25.690414 23154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:43441 every 8 connection(s)
Sep 25 22:54:25 dist-test-slave-x1lh krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376865, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:25.691280 23024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:54:25.693869 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 23024
I20260925 22:54:25.693990 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:54:25.694320 20939 external_mini_cluster.cc:1428] Setting key ed7d57352c504d5a82b0f3e3dcc4f3e4
I20260925 22:54:25.695046 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:25.680809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:41927 (local address 127.20.114.254:42893)
0925 22:54:25.680978 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:25.680983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:25.681841 (+ 858us) server_negotiation.cc:407] Connection header received
0925 22:54:25.683099 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:25.683102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:25.683158 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:25.683253 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:25.684945 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.685878 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.687726 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.687949 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.688077 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.691294 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:25.691299 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:25.691302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:25.691333 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:25.692994 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:25.693472 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:25.693475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:25.693476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:25.693548 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:25.693844 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:25.693848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:25.693851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:25.694109 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:25.694217 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:25.694745 (+ 528us) server_negotiation.cc:299] Negotiation successful
0925 22:54:25.694886 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 22:54:25.695950 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:25.681217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:42893 (local address 127.20.114.195:41927)
0925 22:54:25.681662 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:25.681712 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:25.682810 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:25.683481 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:25.683491 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:25.683994 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:25.684756 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:25.684775 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.686033 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.686036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:25.687574 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:25.687587 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.688141 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.688149 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:25.688398 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.689175 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:25.689194 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:25.691117 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:25.693094 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:25.693099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:25.693110 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:25.693366 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:25.693621 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:25.693623 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:25.693624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:25.693746 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:25.694210 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:25.694215 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:25.694496 (+ 281us) client_negotiation.cc:769] Sending connection context
0925 22:54:25.694682 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 22:54:25.695030 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":124,"threads_started":1}
I20260925 22:54:25.697167 23155 heartbeater.cc:347] Connected to a master server at 127.20.114.254:42893
I20260925 22:54:25.697299 23155 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:25.697861 23155 heartbeater.cc:511] Master 127.20.114.254:42893 requested a full tablet report, sending...
I20260925 22:54:25.698869 22671 ts_manager.cc:194] Registered new tserver with Master: 3ffde459ed8a47379bfe0a0b314656a0 (127.20.114.195:43441)
I20260925 22:54:25.699595 22671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:41927
I20260925 22:54:25.706086 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:25.699595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:40999 (local address 127.20.114.254:42893)
0925 22:54:25.699720 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:25.699724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:25.699916 (+ 192us) server_negotiation.cc:407] Connection header received
0925 22:54:25.700005 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:25.700008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:25.700053 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:25.700138 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:25.700725 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.701265 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.702172 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.702367 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.702444 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.703527 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:25.703529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:25.703531 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:25.703548 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:25.704719 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:25.705144 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:25.705147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:25.705148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:25.705217 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:25.705484 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:25.705486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:25.705487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:25.705631 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:25.705700 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:25.705898 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 22:54:25.705981 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:54:25.706096 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:25.699518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:42893 (local address 127.20.114.195:40999)
0925 22:54:25.699796 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:25.699815 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:25.699920 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:25.700180 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:25.700183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:25.700468 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:25.700619 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:25.700626 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.701399 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.701402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:25.702075 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:25.702084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.702465 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.702468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:25.702536 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.703084 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:25.703098 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:25.703417 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:25.704810 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:25.704813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:25.704814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:25.705063 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:25.705295 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:25.705297 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:25.705298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:25.705413 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:25.705705 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:25.705707 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:25.705777 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 22:54:25.705861 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 22:54:25.705963 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":24,"thread_start_us":96,"threads_started":1}
I20260925 22:54:25.708446 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:54:25 dist-test-slave-x1lh krb5kdc[22627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:54:25 dist-test-slave-x1lh krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:25.733600 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:25.725457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36680 (local address 127.20.114.254:42893)
0925 22:54:25.725618 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:25.725622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:25.725636 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:54:25.725676 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:25.725678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:25.725717 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:25.725811 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:25.726493 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.727020 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.727879 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.728096 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.728209 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.730664 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:25.730668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:25.730670 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:25.730693 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:25.732056 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:25.732504 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:25.732506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:25.732508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:25.732584 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:25.732804 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:25.732806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:25.732807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:25.733042 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:25.733146 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:25.733348 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 22:54:25.733470 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:54:25.736438 22671 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:54:25.739523 22671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:54:25.754266 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:25.748127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:44777 (local address 127.0.0.1:57378)
0925 22:54:25.748793 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:25.748806 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:25.748934 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:25.749451 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:25.749457 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:25.749481 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:25.749636 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:25.749643 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.750959 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.750965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:25.752489 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:25.752500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.753592 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.753596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:25.753716 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.753790 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 22:54:25.753907 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 22:54:25.754066 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":94,"thread_start_us":97,"threads_started":1}
I20260925 22:54:25.754330 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:25.749251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54302 (local address 127.20.114.195:43441)
0925 22:54:25.750000 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:25.750016 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:25.750039 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:54:25.750105 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:25.750109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:25.750280 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:25.750482 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:25.750721 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.751723 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.753106 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.753766 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.753924 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.754048 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:25.754098 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 22:54:25.754195 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":131,"threads_started":1}
I20260925 22:54:25.754268 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:25.747913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:39451 (local address 127.0.0.1:33438)
0925 22:54:25.748630 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:25.748680 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:25.748810 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:25.749689 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:25.749692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:25.749709 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:25.749811 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:25.749821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.751240 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.751243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:25.752498 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:25.752505 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.753611 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.753615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:25.753758 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.753798 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 22:54:25.753905 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 22:54:25.754075 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":138,"thread_start_us":61,"threads_started":1}
I20260925 22:54:25.754268 23176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:25.748905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:43441 (local address 127.0.0.1:54302)
0925 22:54:25.749245 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:25.749259 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:25.749349 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:25.750442 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:25.750445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:25.750463 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:25.750608 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:25.750615 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.751830 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.751836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:25.752989 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:25.752998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.753884 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.753888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:25.753985 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.754008 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:54:25.754064 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 22:54:25.754117 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260925 22:54:25.755084 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:25.748103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33438 (local address 127.20.114.194:39451)
0925 22:54:25.749141 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:25.749155 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:25.749179 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:54:25.749270 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:25.749280 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:25.749561 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:25.749719 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:25.749959 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.751124 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.752629 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.753468 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.754325 (+ 857us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.754825 (+ 500us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:25.754891 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 22:54:25.754980 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":936,"thread_start_us":111,"threads_started":1}
I20260925 22:54:25.755086 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:25.748283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57378 (local address 127.20.114.193:44777)
0925 22:54:25.748904 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:25.748921 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:25.748949 (+ 28us) server_negotiation.cc:407] Connection header received
0925 22:54:25.749033 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:25.749038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:25.749250 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:25.749793 (+ 543us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:25.749937 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.750855 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.752629 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.753468 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.753619 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.753763 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:25.754839 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0925 22:54:25.754949 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":81,"threads_started":1}
I20260925 22:54:25.757637 22800 tablet_service.cc:1467] Processing CreateTablet for tablet bf411b41bade4ad39a32e05cca404163 (DEFAULT_TABLE table=test-table [id=567ccfedd9134ca0bea6b956dae8692c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:25.757758 23089 tablet_service.cc:1467] Processing CreateTablet for tablet bf411b41bade4ad39a32e05cca404163 (DEFAULT_TABLE table=test-table [id=567ccfedd9134ca0bea6b956dae8692c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:25.758100 22949 tablet_service.cc:1467] Processing CreateTablet for tablet bf411b41bade4ad39a32e05cca404163 (DEFAULT_TABLE table=test-table [id=567ccfedd9134ca0bea6b956dae8692c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:25.758646 22800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf411b41bade4ad39a32e05cca404163. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:25.758765 23089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf411b41bade4ad39a32e05cca404163. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:25.758977 22949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf411b41bade4ad39a32e05cca404163. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:25.765007 23178 tablet_bootstrap.cc:492] T bf411b41bade4ad39a32e05cca404163 P a3525409ceea4c7da73004ad65738ffc: Bootstrap starting.
I20260925 22:54:25.766390 23179 tablet_bootstrap.cc:492] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0: Bootstrap starting.
I20260925 22:54:25.767347 23178 tablet_bootstrap.cc:654] T bf411b41bade4ad39a32e05cca404163 P a3525409ceea4c7da73004ad65738ffc: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:25.767359 23180 tablet_bootstrap.cc:492] T bf411b41bade4ad39a32e05cca404163 P 036521f784e04ecca2343b74e9fda6bf: Bootstrap starting.
I20260925 22:54:25.768232 23178 log.cc:824] T bf411b41bade4ad39a32e05cca404163 P a3525409ceea4c7da73004ad65738ffc: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:25.769069 23179 tablet_bootstrap.cc:654] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:25.769546 23180 tablet_bootstrap.cc:654] T bf411b41bade4ad39a32e05cca404163 P 036521f784e04ecca2343b74e9fda6bf: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:25.769883 23179 log.cc:824] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:25.770383 23180 log.cc:824] T bf411b41bade4ad39a32e05cca404163 P 036521f784e04ecca2343b74e9fda6bf: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:25.770433 23178 tablet_bootstrap.cc:492] T bf411b41bade4ad39a32e05cca404163 P a3525409ceea4c7da73004ad65738ffc: No bootstrap required, opened a new log
I20260925 22:54:25.770620 23178 ts_tablet_manager.cc:1397] T bf411b41bade4ad39a32e05cca404163 P a3525409ceea4c7da73004ad65738ffc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:54:25.771775 23179 tablet_bootstrap.cc:492] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0: No bootstrap required, opened a new log
I20260925 22:54:25.771916 23179 ts_tablet_manager.cc:1397] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:54:25.772293 23180 tablet_bootstrap.cc:492] T bf411b41bade4ad39a32e05cca404163 P 036521f784e04ecca2343b74e9fda6bf: No bootstrap required, opened a new log
I20260925 22:54:25.772465 23180 ts_tablet_manager.cc:1397] T bf411b41bade4ad39a32e05cca404163 P 036521f784e04ecca2343b74e9fda6bf: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 22:54:25.774672 23179 raft_consensus.cc:348] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:25.774871 23179 raft_consensus.cc:374] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:25.774693 23178 raft_consensus.cc:348] T bf411b41bade4ad39a32e05cca404163 P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:25.774915 23179 raft_consensus.cc:729] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ffde459ed8a47379bfe0a0b314656a0, State: Initialized, Role: FOLLOWER
I20260925 22:54:25.774946 23178 raft_consensus.cc:374] T bf411b41bade4ad39a32e05cca404163 P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:25.775003 23178 raft_consensus.cc:729] T bf411b41bade4ad39a32e05cca404163 P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3525409ceea4c7da73004ad65738ffc, State: Initialized, Role: FOLLOWER
I20260925 22:54:25.775434 23178 consensus_queue.cc:260] T bf411b41bade4ad39a32e05cca404163 P a3525409ceea4c7da73004ad65738ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:25.775673 23179 consensus_queue.cc:260] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:25.776276 23180 raft_consensus.cc:348] T bf411b41bade4ad39a32e05cca404163 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:25.776510 23178 ts_tablet_manager.cc:1428] T bf411b41bade4ad39a32e05cca404163 P a3525409ceea4c7da73004ad65738ffc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:54:25.776525 23180 raft_consensus.cc:374] T bf411b41bade4ad39a32e05cca404163 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:25.776578 23180 raft_consensus.cc:729] T bf411b41bade4ad39a32e05cca404163 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036521f784e04ecca2343b74e9fda6bf, State: Initialized, Role: FOLLOWER
I20260925 22:54:25.776611 23179 ts_tablet_manager.cc:1428] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:54:25.776731 23155 heartbeater.cc:503] Master 127.20.114.254:42893 was elected leader, sending a full tablet report...
I20260925 22:54:25.776839 22875 heartbeater.cc:503] Master 127.20.114.254:42893 was elected leader, sending a full tablet report...
I20260925 22:54:25.777020 23180 consensus_queue.cc:260] T bf411b41bade4ad39a32e05cca404163 P 036521f784e04ecca2343b74e9fda6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:25.777611 23015 heartbeater.cc:503] Master 127.20.114.254:42893 was elected leader, sending a full tablet report...
I20260925 22:54:25.777887 23180 ts_tablet_manager.cc:1428] T bf411b41bade4ad39a32e05cca404163 P 036521f784e04ecca2343b74e9fda6bf: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260925 22:54:25.942377 23156 tablet.cc:2369] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:54:25.986095 23185 raft_consensus.cc:482] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:54:25.986341 23185 raft_consensus.cc:504] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:25.987648 23185 leader_election.cc:288] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 036521f784e04ecca2343b74e9fda6bf (127.20.114.194:39451), a3525409ceea4c7da73004ad65738ffc (127.20.114.193:44777)
W20260925 22:54:25.988576 23016 tablet.cc:2369] T bf411b41bade4ad39a32e05cca404163 P 036521f784e04ecca2343b74e9fda6bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:54:25.992274 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:25.987830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:39451 (local address 127.20.114.195:42001)
0925 22:54:25.988156 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:25.988173 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:25.988261 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:25.988499 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:25.988502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:25.988519 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:25.988641 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:25.988673 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.989724 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.989728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:25.991002 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:25.991012 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.991848 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.991851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:25.991969 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.991993 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:54:25.992067 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 22:54:25.992138 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":25}
I20260925 22:54:25.992286 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:25.988032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:42001 (local address 127.20.114.194:39451)
0925 22:54:25.988283 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:25.988290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:25.988306 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:54:25.988363 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:25.988366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:25.988410 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:25.988480 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:25.988805 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.989593 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.991139 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.991745 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.991854 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.991903 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:25.992097 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 22:54:25.992171 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260925 22:54:25.992965 22969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf411b41bade4ad39a32e05cca404163" candidate_uuid: "3ffde459ed8a47379bfe0a0b314656a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "036521f784e04ecca2343b74e9fda6bf" is_pre_election: true
I20260925 22:54:25.993124 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:25.988151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:39719 (local address 127.20.114.193:44777)
0925 22:54:25.988278 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:25.988283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:25.988397 (+ 114us) server_negotiation.cc:407] Connection header received
0925 22:54:25.989101 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:25.989105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:25.989147 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:25.989191 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:25.989999 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.990493 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.991737 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.992338 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.992477 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.992536 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:25.992919 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 22:54:25.993009 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:54:25.993217 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:25.987951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:44777 (local address 127.20.114.195:39719)
0925 22:54:25.988320 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:25.988333 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:25.988962 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:25.989324 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:25.989326 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:25.989341 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:25.989886 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:25.989894 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.990602 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.990609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:25.991599 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:25.991608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:25.992632 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:25.992635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:25.992737 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:25.992760 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:54:25.992881 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 22:54:25.992947 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":68,"thread_start_us":114,"threads_started":1}
I20260925 22:54:25.993286 22969 raft_consensus.cc:2454] T bf411b41bade4ad39a32e05cca404163 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ffde459ed8a47379bfe0a0b314656a0 in term 0.
I20260925 22:54:25.993767 23042 leader_election.cc:302] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 036521f784e04ecca2343b74e9fda6bf, 3ffde459ed8a47379bfe0a0b314656a0; no voters:
I20260925 22:54:25.993775 22829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf411b41bade4ad39a32e05cca404163" candidate_uuid: "3ffde459ed8a47379bfe0a0b314656a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3525409ceea4c7da73004ad65738ffc" is_pre_election: true
I20260925 22:54:25.994050 22829 raft_consensus.cc:2454] T bf411b41bade4ad39a32e05cca404163 P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ffde459ed8a47379bfe0a0b314656a0 in term 0.
I20260925 22:54:25.994030 23185 raft_consensus.cc:2781] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:54:25.994130 23185 raft_consensus.cc:482] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:54:25.994174 23185 raft_consensus.cc:3037] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:25.995298 23185 raft_consensus.cc:504] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:25.995664 23185 leader_election.cc:288] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0 [CANDIDATE]: Term 1 election: Requested vote from peers 036521f784e04ecca2343b74e9fda6bf (127.20.114.194:39451), a3525409ceea4c7da73004ad65738ffc (127.20.114.193:44777)
I20260925 22:54:25.995995 22829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf411b41bade4ad39a32e05cca404163" candidate_uuid: "3ffde459ed8a47379bfe0a0b314656a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3525409ceea4c7da73004ad65738ffc"
I20260925 22:54:25.996088 22829 raft_consensus.cc:3037] T bf411b41bade4ad39a32e05cca404163 P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:25.997255 22829 raft_consensus.cc:2454] T bf411b41bade4ad39a32e05cca404163 P a3525409ceea4c7da73004ad65738ffc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ffde459ed8a47379bfe0a0b314656a0 in term 1.
I20260925 22:54:25.997521 22969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf411b41bade4ad39a32e05cca404163" candidate_uuid: "3ffde459ed8a47379bfe0a0b314656a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "036521f784e04ecca2343b74e9fda6bf"
I20260925 22:54:25.997560 23043 leader_election.cc:302] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ffde459ed8a47379bfe0a0b314656a0, a3525409ceea4c7da73004ad65738ffc; no voters:
I20260925 22:54:25.997666 22969 raft_consensus.cc:3037] T bf411b41bade4ad39a32e05cca404163 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:25.997730 23185 raft_consensus.cc:2781] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:25.997989 23185 raft_consensus.cc:686] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0 [term 1 LEADER]: Becoming Leader. State: Replica: 3ffde459ed8a47379bfe0a0b314656a0, State: Running, Role: LEADER
I20260925 22:54:25.998322 23185 consensus_queue.cc:237] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:25.998905 22969 raft_consensus.cc:2454] T bf411b41bade4ad39a32e05cca404163 P 036521f784e04ecca2343b74e9fda6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ffde459ed8a47379bfe0a0b314656a0 in term 1.
I20260925 22:54:26.001227 22670 catalog_manager.cc:5495] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ffde459ed8a47379bfe0a0b314656a0 (127.20.114.195). New cstate: current_term: 1 leader_uuid: "3ffde459ed8a47379bfe0a0b314656a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } health_report { overall_health: HEALTHY } } }
W20260925 22:54:26.014496 22876 tablet.cc:2369] T bf411b41bade4ad39a32e05cca404163 P a3525409ceea4c7da73004ad65738ffc: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 22:54:26 dist-test-slave-x1lh krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376864, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:26.071149 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:26.062642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36692 (local address 127.20.114.254:42893)
0925 22:54:26.062808 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:26.062813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:26.063076 (+ 263us) server_negotiation.cc:407] Connection header received
0925 22:54:26.063173 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:26.063175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:26.063216 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:26.063297 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:26.064007 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.064514 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.065398 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.065603 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.065683 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:26.067979 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:26.067983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:26.067985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:26.068003 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:26.069484 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:26.069955 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:26.069958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:26.069960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:26.070033 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:26.070296 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:26.070299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:26.070301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:26.070462 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:26.070559 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:26.070840 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 22:54:26.070997 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":90}
I20260925 22:54:26.071163 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:26.062551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:42893 (local address 127.0.0.1:36692)
0925 22:54:26.062970 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:26.062983 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:26.063093 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:26.063329 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:26.063331 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:26.063796 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:26.063891 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:26.063897 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.064605 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.064607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:26.065295 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:26.065304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.065698 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.065700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:26.065761 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:26.066344 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:26.066387 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:26.067817 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:26.069603 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:26.069612 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:26.069619 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:26.069841 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:26.070125 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:26.070127 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:26.070130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:26.070208 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:26.070568 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:26.070577 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:26.070682 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 22:54:26.070796 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 22:54:26.070944 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":82,"threads_started":1}
I20260925 22:54:26.075024 22670 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:36692:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 22:54:26.075724 22670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:54:26.080464 22800 tablet_service.cc:1467] Processing CreateTablet for tablet 2ad916d74eaf4522bc7606fa28d3d5e5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac83c94879914f7988f9dcb22ddbfb75]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:54:26.080595 23089 tablet_service.cc:1467] Processing CreateTablet for tablet 2ad916d74eaf4522bc7606fa28d3d5e5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac83c94879914f7988f9dcb22ddbfb75]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:54:26.080580 22949 tablet_service.cc:1467] Processing CreateTablet for tablet 2ad916d74eaf4522bc7606fa28d3d5e5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac83c94879914f7988f9dcb22ddbfb75]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:54:26.081236 23089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ad916d74eaf4522bc7606fa28d3d5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:26.081238 22800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ad916d74eaf4522bc7606fa28d3d5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:26.081311 22949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ad916d74eaf4522bc7606fa28d3d5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:26.083897 23180 tablet_bootstrap.cc:492] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf: Bootstrap starting.
I20260925 22:54:26.084029 23178 tablet_bootstrap.cc:492] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc: Bootstrap starting.
I20260925 22:54:26.084297 23179 tablet_bootstrap.cc:492] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 3ffde459ed8a47379bfe0a0b314656a0: Bootstrap starting.
I20260925 22:54:26.085018 23180 tablet_bootstrap.cc:654] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:26.085029 23178 tablet_bootstrap.cc:654] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:26.085371 23179 tablet_bootstrap.cc:654] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 3ffde459ed8a47379bfe0a0b314656a0: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:26.086395 23178 tablet_bootstrap.cc:492] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc: No bootstrap required, opened a new log
I20260925 22:54:26.086484 23178 ts_tablet_manager.cc:1397] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 22:54:26.086629 23180 tablet_bootstrap.cc:492] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf: No bootstrap required, opened a new log
I20260925 22:54:26.086709 23180 ts_tablet_manager.cc:1397] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260925 22:54:26.086730 23179 tablet_bootstrap.cc:492] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 3ffde459ed8a47379bfe0a0b314656a0: No bootstrap required, opened a new log
I20260925 22:54:26.086812 23179 ts_tablet_manager.cc:1397] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 3ffde459ed8a47379bfe0a0b314656a0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 22:54:26.086957 23178 raft_consensus.cc:348] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:26.087074 23178 raft_consensus.cc:374] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:26.087118 23178 raft_consensus.cc:729] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3525409ceea4c7da73004ad65738ffc, State: Initialized, Role: FOLLOWER
I20260925 22:54:26.087139 23180 raft_consensus.cc:348] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:26.087241 23180 raft_consensus.cc:374] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:26.087282 23180 raft_consensus.cc:729] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036521f784e04ecca2343b74e9fda6bf, State: Initialized, Role: FOLLOWER
I20260925 22:54:26.087257 23178 consensus_queue.cc:260] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:26.087245 23179 raft_consensus.cc:348] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:26.087380 23179 raft_consensus.cc:374] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:26.087428 23179 raft_consensus.cc:729] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ffde459ed8a47379bfe0a0b314656a0, State: Initialized, Role: FOLLOWER
I20260925 22:54:26.087414 23180 consensus_queue.cc:260] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:26.087533 23178 ts_tablet_manager.cc:1428] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:54:26.087586 23179 consensus_queue.cc:260] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 3ffde459ed8a47379bfe0a0b314656a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:26.087663 23186 tablet_replica.cc:439] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } } }
I20260925 22:54:26.087733 23184 tablet_replica.cc:439] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } } }
I20260925 22:54:26.087803 23186 tablet_replica.cc:442] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:54:26.087796 23185 tablet_replica.cc:439] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 3ffde459ed8a47379bfe0a0b314656a0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } } }
I20260925 22:54:26.087859 23184 tablet_replica.cc:442] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:54:26.087901 23185 tablet_replica.cc:442] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 3ffde459ed8a47379bfe0a0b314656a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:54:26.088096 23179 ts_tablet_manager.cc:1428] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 3ffde459ed8a47379bfe0a0b314656a0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:54:26.087667 23180 ts_tablet_manager.cc:1428] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:54:26.204452 23186 raft_consensus.cc:482] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:54:26.204596 23186 raft_consensus.cc:504] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:26.205770 23186 leader_election.cc:288] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3525409ceea4c7da73004ad65738ffc (127.20.114.193:44777), 3ffde459ed8a47379bfe0a0b314656a0 (127.20.114.195:43441)
I20260925 22:54:26.210421 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:26.206180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:33627 (local address 127.20.114.195:43441)
0925 22:54:26.206314 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:26.206319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:26.206547 (+ 228us) server_negotiation.cc:407] Connection header received
0925 22:54:26.206656 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:26.206659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:26.206710 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:26.206792 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:26.207089 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.207838 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.209183 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.209852 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.209953 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:26.209994 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:26.210216 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 22:54:26.210269 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:54:26.210496 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:26.206075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:43441 (local address 127.20.114.194:33627)
0925 22:54:26.206442 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:26.206467 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:26.206572 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:26.206810 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:26.206813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:26.206828 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:26.206971 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:26.206979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.207934 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.207938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:26.209068 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:26.209077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.209982 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.209986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:26.210088 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:26.210114 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:54:26.210197 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 22:54:26.210271 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":86,"threads_started":1}
I20260925 22:54:26.210857 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:26.206185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:44057 (local address 127.20.114.193:44777)
0925 22:54:26.206415 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:26.206420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:26.206662 (+ 242us) server_negotiation.cc:407] Connection header received
0925 22:54:26.206774 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:26.206779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:26.206830 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:26.206929 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:26.207654 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.208402 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.209638 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.210234 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.210346 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:26.210391 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:26.210609 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 22:54:26.210722 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260925 22:54:26.211064 23109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad916d74eaf4522bc7606fa28d3d5e5" candidate_uuid: "036521f784e04ecca2343b74e9fda6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ffde459ed8a47379bfe0a0b314656a0" is_pre_election: true
I20260925 22:54:26.211227 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:26.206075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:44777 (local address 127.20.114.194:44057)
0925 22:54:26.206553 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:26.206567 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:26.206674 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:26.206950 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:26.206952 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:26.206964 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:26.207512 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:26.207521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.208512 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.208514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:26.209506 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:26.209515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.210339 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.210343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:26.210452 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:26.210477 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:54:26.211007 (+ 530us) client_negotiation.cc:240] Negotiation successful
0925 22:54:26.211078 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":47,"thread_start_us":86,"threads_started":1}
I20260925 22:54:26.211274 23109 raft_consensus.cc:2454] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 036521f784e04ecca2343b74e9fda6bf in term 0.
I20260925 22:54:26.211585 23184 raft_consensus.cc:482] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:54:26.211686 23184 raft_consensus.cc:504] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:26.211756 22829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad916d74eaf4522bc7606fa28d3d5e5" candidate_uuid: "036521f784e04ecca2343b74e9fda6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3525409ceea4c7da73004ad65738ffc" is_pre_election: true
I20260925 22:54:26.211778 22902 leader_election.cc:302] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 036521f784e04ecca2343b74e9fda6bf, 3ffde459ed8a47379bfe0a0b314656a0; no voters:
I20260925 22:54:26.212050 23186 raft_consensus.cc:2781] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:54:26.212077 22829 raft_consensus.cc:2454] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 036521f784e04ecca2343b74e9fda6bf in term 0.
I20260925 22:54:26.212118 23186 raft_consensus.cc:482] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:54:26.212167 23186 raft_consensus.cc:3037] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:26.212805 23184 leader_election.cc:288] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 036521f784e04ecca2343b74e9fda6bf (127.20.114.194:39451), 3ffde459ed8a47379bfe0a0b314656a0 (127.20.114.195:43441)
I20260925 22:54:26.213474 23186 raft_consensus.cc:504] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:26.213865 23186 leader_election.cc:288] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf [CANDIDATE]: Term 1 election: Requested vote from peers a3525409ceea4c7da73004ad65738ffc (127.20.114.193:44777), 3ffde459ed8a47379bfe0a0b314656a0 (127.20.114.195:43441)
I20260925 22:54:26.214139 22829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad916d74eaf4522bc7606fa28d3d5e5" candidate_uuid: "036521f784e04ecca2343b74e9fda6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3525409ceea4c7da73004ad65738ffc"
I20260925 22:54:26.214195 23109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad916d74eaf4522bc7606fa28d3d5e5" candidate_uuid: "036521f784e04ecca2343b74e9fda6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ffde459ed8a47379bfe0a0b314656a0"
I20260925 22:54:26.214255 22829 raft_consensus.cc:3037] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:26.214344 23109 raft_consensus.cc:3037] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:26.215369 22829 raft_consensus.cc:2454] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 036521f784e04ecca2343b74e9fda6bf in term 1.
I20260925 22:54:26.215601 23109 raft_consensus.cc:2454] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 3ffde459ed8a47379bfe0a0b314656a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 036521f784e04ecca2343b74e9fda6bf in term 1.
I20260925 22:54:26.215726 22903 leader_election.cc:302] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 036521f784e04ecca2343b74e9fda6bf, a3525409ceea4c7da73004ad65738ffc; no voters:
I20260925 22:54:26.215934 23186 raft_consensus.cc:2781] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:26.216099 23186 raft_consensus.cc:686] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf [term 1 LEADER]: Becoming Leader. State: Replica: 036521f784e04ecca2343b74e9fda6bf, State: Running, Role: LEADER
I20260925 22:54:26.216427 23186 consensus_queue.cc:237] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:26.218415 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:26.213184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:50625 (local address 127.20.114.194:39451)
0925 22:54:26.213302 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:26.213306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:26.213507 (+ 201us) server_negotiation.cc:407] Connection header received
0925 22:54:26.213673 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:26.213676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:26.213724 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:26.213760 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:26.214506 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.215159 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.217246 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.217821 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.217925 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:26.217966 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:26.218229 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 22:54:26.218301 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:54:26.218428 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:26.213787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:35997 (local address 127.20.114.195:43441)
0925 22:54:26.213901 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:26.213905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:26.214204 (+ 299us) server_negotiation.cc:407] Connection header received
0925 22:54:26.214378 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:26.214381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:26.214426 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:26.214472 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:26.215263 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.215926 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.217254 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.217821 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.217921 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:26.217966 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:26.218278 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 22:54:26.218339 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 22:54:26.218811 23202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:26.213458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:43441 (local address 127.20.114.193:35997)
0925 22:54:26.214124 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:26.214139 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:26.214264 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:26.214601 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:26.214604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:26.214622 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:26.215139 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:26.215147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.216050 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.216053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:26.217124 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:26.217133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.218061 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.218064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:26.218137 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:26.218151 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 22:54:26.218587 (+ 436us) client_negotiation.cc:240] Negotiation successful
0925 22:54:26.218672 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":48,"thread_start_us":118,"threads_started":1}
I20260925 22:54:26.219228 23109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad916d74eaf4522bc7606fa28d3d5e5" candidate_uuid: "a3525409ceea4c7da73004ad65738ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ffde459ed8a47379bfe0a0b314656a0" is_pre_election: true
I20260925 22:54:26.219439 23109 raft_consensus.cc:2379] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 3ffde459ed8a47379bfe0a0b314656a0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a3525409ceea4c7da73004ad65738ffc in current term 1: Already voted for candidate 036521f784e04ecca2343b74e9fda6bf in this term.
I20260925 22:54:26.219462 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:26.213118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:39451 (local address 127.20.114.193:50625)
0925 22:54:26.213423 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:26.213435 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:26.213544 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:26.214131 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:26.214134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:26.214151 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:26.214333 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:26.214342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.215300 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.215305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:26.217136 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:26.217147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.217986 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.217990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:26.218090 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:26.218115 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:54:26.218203 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 22:54:26.218275 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":66,"thread_start_us":84,"threads_started":1}
I20260925 22:54:26.219822 23203 tablet_replica.cc:439] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf: TxnStatusTablet state changed. Reason: New leader 036521f784e04ecca2343b74e9fda6bf. Latest consensus state: current_term: 1 leader_uuid: "036521f784e04ecca2343b74e9fda6bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } } }
I20260925 22:54:26.219930 23203 tablet_replica.cc:442] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:54:26.220139 22969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad916d74eaf4522bc7606fa28d3d5e5" candidate_uuid: "a3525409ceea4c7da73004ad65738ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "036521f784e04ecca2343b74e9fda6bf" is_pre_election: true
I20260925 22:54:26.220558 22762 leader_election.cc:302] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3525409ceea4c7da73004ad65738ffc; no voters: 036521f784e04ecca2343b74e9fda6bf, 3ffde459ed8a47379bfe0a0b314656a0
I20260925 22:54:26.220561 22670 catalog_manager.cc:5495] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 036521f784e04ecca2343b74e9fda6bf (127.20.114.194). New cstate: current_term: 1 leader_uuid: "036521f784e04ecca2343b74e9fda6bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } health_report { overall_health: UNKNOWN } } }
I20260925 22:54:26.220780 23184 raft_consensus.cc:2726] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 22:54:26.221333 23205 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 22:54:26.221541 23205 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 22:54:26.240053 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:26.236028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33440 (local address 127.20.114.194:39451)
0925 22:54:26.236146 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:26.236149 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:26.236205 (+ 56us) server_negotiation.cc:407] Connection header received
0925 22:54:26.236423 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:26.236425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:26.236476 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:26.236556 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:26.236837 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.237410 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.238612 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.238818 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.238891 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:26.239232 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:26.239677 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:26.239762 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:26.239883 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 22:54:26.239929 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 22:54:26.240085 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:26.235928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:39451 (local address 127.0.0.1:33440)
0925 22:54:26.236147 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:26.236161 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:26.236270 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:26.236592 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:26.236596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:26.236604 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:54:26.236728 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:26.236734 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.237515 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.237519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:26.238510 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:26.238522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.238918 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.238921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:26.239018 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:26.239070 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:54:26.239770 (+ 700us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:54:26.239785 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 22:54:26.239866 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 22:54:26.239948 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260925 22:54:26.245503 23109 raft_consensus.cc:1261] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 3ffde459ed8a47379bfe0a0b314656a0 [term 1 FOLLOWER]: Refusing update from remote peer 036521f784e04ecca2343b74e9fda6bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:26.245620 22829 raft_consensus.cc:1261] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc [term 1 FOLLOWER]: Refusing update from remote peer 036521f784e04ecca2343b74e9fda6bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:26.246099 23203 consensus_queue.cc:1035] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:26.246286 23186 consensus_queue.cc:1035] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:26.250084 23184 tablet_replica.cc:439] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc: TxnStatusTablet state changed. Reason: New leader 036521f784e04ecca2343b74e9fda6bf. Latest consensus state: current_term: 1 leader_uuid: "036521f784e04ecca2343b74e9fda6bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } } }
I20260925 22:54:26.250202 23184 tablet_replica.cc:442] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:54:26.251040 23185 tablet_replica.cc:439] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 3ffde459ed8a47379bfe0a0b314656a0: TxnStatusTablet state changed. Reason: New leader 036521f784e04ecca2343b74e9fda6bf. Latest consensus state: current_term: 1 leader_uuid: "036521f784e04ecca2343b74e9fda6bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } } }
I20260925 22:54:26.251142 23185 tablet_replica.cc:442] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 3ffde459ed8a47379bfe0a0b314656a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:54:26.253582 23203 tablet_replica.cc:439] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "036521f784e04ecca2343b74e9fda6bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } } }
I20260925 22:54:26.253691 23203 tablet_replica.cc:442] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:54:26.253798 23203 tablet_replica.cc:439] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "036521f784e04ecca2343b74e9fda6bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } } }
I20260925 22:54:26.253860 23203 tablet_replica.cc:442] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 036521f784e04ecca2343b74e9fda6bf: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:54:26.254158 23184 tablet_replica.cc:439] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "036521f784e04ecca2343b74e9fda6bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } } }
I20260925 22:54:26.254221 23209 mvcc.cc:205] Tried to move back new op lower bound from 7333383644135276544 to 7333383644026122240. Current Snapshot: MvccSnapshot[applied={T|T < 7333383644135276544}]
I20260925 22:54:26.254264 23184 tablet_replica.cc:442] T 2ad916d74eaf4522bc7606fa28d3d5e5 P a3525409ceea4c7da73004ad65738ffc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:54:26.255048 23185 tablet_replica.cc:439] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 3ffde459ed8a47379bfe0a0b314656a0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "036521f784e04ecca2343b74e9fda6bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } } }
I20260925 22:54:26.255163 23185 tablet_replica.cc:442] T 2ad916d74eaf4522bc7606fa28d3d5e5 P 3ffde459ed8a47379bfe0a0b314656a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:54:26.264827 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:26.260478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54318 (local address 127.20.114.195:43441)
0925 22:54:26.260649 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:26.260652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:26.260757 (+ 105us) server_negotiation.cc:407] Connection header received
0925 22:54:26.260966 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:26.260969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:26.261040 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:26.261145 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:26.261412 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.261982 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.263311 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.263632 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.263736 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:26.264012 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:26.264464 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:26.264565 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:26.264655 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 22:54:26.264705 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260925 22:54:26.276890 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:26.273030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:34131 (local address 127.20.114.194:39451)
0925 22:54:26.273140 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:26.273145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:26.273158 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:54:26.273207 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:26.273210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:26.273259 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:26.273342 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:26.273602 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.274376 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.275571 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.275922 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.275988 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:26.276407 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:26.276493 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:26.276568 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:26.276671 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 22:54:26.276727 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:54:26.276944 23221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:26.272647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:39451 (local address 127.20.114.195:34131)
0925 22:54:26.272954 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:26.272970 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:26.273064 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:26.273355 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:26.273358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:26.273365 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:54:26.273493 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:26.273498 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.274483 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.274486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:26.275436 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:26.275446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.276096 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.276100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:26.276192 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:26.276239 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:54:26.276581 (+ 342us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:54:26.276587 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:54:26.276648 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 22:54:26.276721 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":93,"threads_started":1}
I20260925 22:54:26.283607 22829 raft_consensus.cc:1261] T bf411b41bade4ad39a32e05cca404163 P a3525409ceea4c7da73004ad65738ffc [term 1 FOLLOWER]: Refusing update from remote peer 3ffde459ed8a47379bfe0a0b314656a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:26.283900 22969 raft_consensus.cc:1261] T bf411b41bade4ad39a32e05cca404163 P 036521f784e04ecca2343b74e9fda6bf [term 1 FOLLOWER]: Refusing update from remote peer 3ffde459ed8a47379bfe0a0b314656a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:26.284186 23188 consensus_queue.cc:1035] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:26.284337 23185 consensus_queue.cc:1035] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:26.291551 23208 mvcc.cc:205] Tried to move back new op lower bound from 7333383644293775360 to 7333383643132055552. Current Snapshot: MvccSnapshot[applied={T|T < 7333383644293775360}]
I20260925 22:54:26.308590 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:26.304462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:43441 (local address 127.20.114.194:52599)
0925 22:54:26.305104 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:26.305120 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:26.305222 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:26.305466 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:26.305469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:26.305477 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:54:26.305627 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:26.305633 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.306532 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.306538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:26.307421 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:26.307431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.307875 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.307886 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:26.307976 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:26.308023 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:54:26.308304 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:54:26.308312 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 22:54:26.308376 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 22:54:26.308449 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":100,"threads_started":1}
I20260925 22:54:26.308571 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:26.304525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:52599 (local address 127.20.114.195:43441)
0925 22:54:26.304638 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:26.304642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:26.305188 (+ 546us) server_negotiation.cc:407] Connection header received
0925 22:54:26.305310 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:26.305312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:26.305350 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:26.305434 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:26.305762 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.306425 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.307536 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.307770 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.307876 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:26.308149 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:26.308223 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:26.308300 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:26.308393 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 22:54:26.308438 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:54:26.334537 22670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:26.334728 22670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:26.337030 22670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d: Sending DeleteTablet for 3 replicas of tablet bf411b41bade4ad39a32e05cca404163
I20260925 22:54:26.337666 22949 tablet_service.cc:1514] Processing DeleteTablet for tablet bf411b41bade4ad39a32e05cca404163 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:26 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:33438
I20260925 22:54:26.337916 22800 tablet_service.cc:1514] Processing DeleteTablet for tablet bf411b41bade4ad39a32e05cca404163 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:26 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:57378
I20260925 22:54:26.338351 23230 tablet_replica.cc:330] T bf411b41bade4ad39a32e05cca404163 P a3525409ceea4c7da73004ad65738ffc: stopping tablet replica
I20260925 22:54:26.338579 23089 tablet_service.cc:1514] Processing DeleteTablet for tablet bf411b41bade4ad39a32e05cca404163 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:26 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:54302
I20260925 22:54:26.338798 23230 raft_consensus.cc:2229] T bf411b41bade4ad39a32e05cca404163 P a3525409ceea4c7da73004ad65738ffc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:26.339048 23230 raft_consensus.cc:2258] T bf411b41bade4ad39a32e05cca404163 P a3525409ceea4c7da73004ad65738ffc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:26.339643 22670 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:54:26.340049 22670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:54:26.340683 23230 ts_tablet_manager.cc:1905] T bf411b41bade4ad39a32e05cca404163 P a3525409ceea4c7da73004ad65738ffc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:26.343142 23231 tablet_replica.cc:330] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0: stopping tablet replica
I20260925 22:54:26.343416 23231 raft_consensus.cc:2229] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:54:26.344010 23230 ts_tablet_manager.cc:1918] T bf411b41bade4ad39a32e05cca404163 P a3525409ceea4c7da73004ad65738ffc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:54:26.344043 23231 raft_consensus.cc:2258] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:26.344172 23230 log.cc:1196] T bf411b41bade4ad39a32e05cca404163 P a3525409ceea4c7da73004ad65738ffc: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-0/wal/wals/bf411b41bade4ad39a32e05cca404163
I20260925 22:54:26.344568 23230 ts_tablet_manager.cc:1939] T bf411b41bade4ad39a32e05cca404163 P a3525409ceea4c7da73004ad65738ffc: Deleting consensus metadata
I20260925 22:54:26.344924 22800 tablet_service.cc:1467] Processing CreateTablet for tablet 93b4c7288a28488ca34c8adba6fd2e8a (DEFAULT_TABLE table=test-table [id=892f5394c71a4673a63e0e21dd9ca404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:26.345184 22800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93b4c7288a28488ca34c8adba6fd2e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:26.345333 22949 tablet_service.cc:1467] Processing CreateTablet for tablet 93b4c7288a28488ca34c8adba6fd2e8a (DEFAULT_TABLE table=test-table [id=892f5394c71a4673a63e0e21dd9ca404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:26.345597 22949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93b4c7288a28488ca34c8adba6fd2e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:26.346077 22659 catalog_manager.cc:4841] TS a3525409ceea4c7da73004ad65738ffc (127.20.114.193:44777): tablet bf411b41bade4ad39a32e05cca404163 (table test-table [id=567ccfedd9134ca0bea6b956dae8692c]) successfully deleted
I20260925 22:54:26.347316 23178 tablet_bootstrap.cc:492] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc: Bootstrap starting.
I20260925 22:54:26.347958 23089 tablet_service.cc:1467] Processing CreateTablet for tablet 93b4c7288a28488ca34c8adba6fd2e8a (DEFAULT_TABLE table=test-table [id=892f5394c71a4673a63e0e21dd9ca404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:26.348194 23089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93b4c7288a28488ca34c8adba6fd2e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:26.348240 23178 tablet_bootstrap.cc:654] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:26.349692 23178 tablet_bootstrap.cc:492] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc: No bootstrap required, opened a new log
I20260925 22:54:26.349771 23178 ts_tablet_manager.cc:1397] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 22:54:26.350170 23178 raft_consensus.cc:348] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:26.350272 23178 raft_consensus.cc:374] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:26.350314 23178 raft_consensus.cc:729] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3525409ceea4c7da73004ad65738ffc, State: Initialized, Role: FOLLOWER
I20260925 22:54:26.350447 23178 consensus_queue.cc:260] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:26.350672 23180 tablet_bootstrap.cc:492] T 93b4c7288a28488ca34c8adba6fd2e8a P 036521f784e04ecca2343b74e9fda6bf: Bootstrap starting.
I20260925 22:54:26.350706 23178 ts_tablet_manager.cc:1428] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:54:26.351420 23180 tablet_bootstrap.cc:654] T 93b4c7288a28488ca34c8adba6fd2e8a P 036521f784e04ecca2343b74e9fda6bf: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:26.352087 23179 tablet_bootstrap.cc:492] T 93b4c7288a28488ca34c8adba6fd2e8a P 3ffde459ed8a47379bfe0a0b314656a0: Bootstrap starting.
I20260925 22:54:26.352962 23179 tablet_bootstrap.cc:654] T 93b4c7288a28488ca34c8adba6fd2e8a P 3ffde459ed8a47379bfe0a0b314656a0: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:26.354090 23180 tablet_bootstrap.cc:492] T 93b4c7288a28488ca34c8adba6fd2e8a P 036521f784e04ecca2343b74e9fda6bf: No bootstrap required, opened a new log
I20260925 22:54:26.354180 23180 ts_tablet_manager.cc:1397] T 93b4c7288a28488ca34c8adba6fd2e8a P 036521f784e04ecca2343b74e9fda6bf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260925 22:54:26.354626 23180 raft_consensus.cc:348] T 93b4c7288a28488ca34c8adba6fd2e8a P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:26.354728 23180 raft_consensus.cc:374] T 93b4c7288a28488ca34c8adba6fd2e8a P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:26.354759 23180 raft_consensus.cc:729] T 93b4c7288a28488ca34c8adba6fd2e8a P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036521f784e04ecca2343b74e9fda6bf, State: Initialized, Role: FOLLOWER
I20260925 22:54:26.354873 23180 consensus_queue.cc:260] T 93b4c7288a28488ca34c8adba6fd2e8a P 036521f784e04ecca2343b74e9fda6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:26.355149 23180 ts_tablet_manager.cc:1428] T 93b4c7288a28488ca34c8adba6fd2e8a P 036521f784e04ecca2343b74e9fda6bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:26.356415 23231 ts_tablet_manager.cc:1905] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:26.359110 23179 tablet_bootstrap.cc:492] T 93b4c7288a28488ca34c8adba6fd2e8a P 3ffde459ed8a47379bfe0a0b314656a0: No bootstrap required, opened a new log
I20260925 22:54:26.359205 23179 ts_tablet_manager.cc:1397] T 93b4c7288a28488ca34c8adba6fd2e8a P 3ffde459ed8a47379bfe0a0b314656a0: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260925 22:54:26.359398 23229 tablet_replica.cc:330] T bf411b41bade4ad39a32e05cca404163 P 036521f784e04ecca2343b74e9fda6bf: stopping tablet replica
I20260925 22:54:26.359648 23179 raft_consensus.cc:348] T 93b4c7288a28488ca34c8adba6fd2e8a P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:26.359763 23179 raft_consensus.cc:374] T 93b4c7288a28488ca34c8adba6fd2e8a P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:26.359802 23179 raft_consensus.cc:729] T 93b4c7288a28488ca34c8adba6fd2e8a P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ffde459ed8a47379bfe0a0b314656a0, State: Initialized, Role: FOLLOWER
I20260925 22:54:26.359954 23179 consensus_queue.cc:260] T 93b4c7288a28488ca34c8adba6fd2e8a P 3ffde459ed8a47379bfe0a0b314656a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:26.360098 23231 ts_tablet_manager.cc:1918] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:54:26.360206 23231 log.cc:1196] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/bf411b41bade4ad39a32e05cca404163
I20260925 22:54:26.359715 23229 raft_consensus.cc:2229] T bf411b41bade4ad39a32e05cca404163 P 036521f784e04ecca2343b74e9fda6bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:26.360487 23229 raft_consensus.cc:2258] T bf411b41bade4ad39a32e05cca404163 P 036521f784e04ecca2343b74e9fda6bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:26.361135 23231 ts_tablet_manager.cc:1939] T bf411b41bade4ad39a32e05cca404163 P 3ffde459ed8a47379bfe0a0b314656a0: Deleting consensus metadata
I20260925 22:54:26.360234 23179 ts_tablet_manager.cc:1428] T 93b4c7288a28488ca34c8adba6fd2e8a P 3ffde459ed8a47379bfe0a0b314656a0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:54:26.362367 22658 catalog_manager.cc:4841] TS 3ffde459ed8a47379bfe0a0b314656a0 (127.20.114.195:43441): tablet bf411b41bade4ad39a32e05cca404163 (table test-table [id=567ccfedd9134ca0bea6b956dae8692c]) successfully deleted
I20260925 22:54:26.363149 23229 ts_tablet_manager.cc:1905] T bf411b41bade4ad39a32e05cca404163 P 036521f784e04ecca2343b74e9fda6bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:26.366204 23229 ts_tablet_manager.cc:1918] T bf411b41bade4ad39a32e05cca404163 P 036521f784e04ecca2343b74e9fda6bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:54:26.366295 23229 log.cc:1196] T bf411b41bade4ad39a32e05cca404163 P 036521f784e04ecca2343b74e9fda6bf: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/bf411b41bade4ad39a32e05cca404163
I20260925 22:54:26.366626 23229 ts_tablet_manager.cc:1939] T bf411b41bade4ad39a32e05cca404163 P 036521f784e04ecca2343b74e9fda6bf: Deleting consensus metadata
I20260925 22:54:26.367239 23184 raft_consensus.cc:482] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:54:26.367348 23184 raft_consensus.cc:504] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:26.367784 23184 leader_election.cc:288] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 036521f784e04ecca2343b74e9fda6bf (127.20.114.194:39451), 3ffde459ed8a47379bfe0a0b314656a0 (127.20.114.195:43441)
I20260925 22:54:26.368161 22969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b4c7288a28488ca34c8adba6fd2e8a" candidate_uuid: "a3525409ceea4c7da73004ad65738ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "036521f784e04ecca2343b74e9fda6bf" is_pre_election: true
I20260925 22:54:26.368286 22969 raft_consensus.cc:2454] T 93b4c7288a28488ca34c8adba6fd2e8a P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3525409ceea4c7da73004ad65738ffc in term 0.
I20260925 22:54:26.368520 23109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b4c7288a28488ca34c8adba6fd2e8a" candidate_uuid: "a3525409ceea4c7da73004ad65738ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ffde459ed8a47379bfe0a0b314656a0" is_pre_election: true
I20260925 22:54:26.368644 23109 raft_consensus.cc:2454] T 93b4c7288a28488ca34c8adba6fd2e8a P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3525409ceea4c7da73004ad65738ffc in term 0.
I20260925 22:54:26.369005 22762 leader_election.cc:302] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 036521f784e04ecca2343b74e9fda6bf, a3525409ceea4c7da73004ad65738ffc; no voters:
I20260925 22:54:26.369170 23184 raft_consensus.cc:2781] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:54:26.369233 23184 raft_consensus.cc:482] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:54:26.369280 23184 raft_consensus.cc:3037] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:26.370826 23184 raft_consensus.cc:504] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:26.371191 23184 leader_election.cc:288] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc [CANDIDATE]: Term 1 election: Requested vote from peers 036521f784e04ecca2343b74e9fda6bf (127.20.114.194:39451), 3ffde459ed8a47379bfe0a0b314656a0 (127.20.114.195:43441)
I20260925 22:54:26.371507 22969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b4c7288a28488ca34c8adba6fd2e8a" candidate_uuid: "a3525409ceea4c7da73004ad65738ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "036521f784e04ecca2343b74e9fda6bf"
I20260925 22:54:26.371636 22969 raft_consensus.cc:3037] T 93b4c7288a28488ca34c8adba6fd2e8a P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:26.372658 22969 raft_consensus.cc:2454] T 93b4c7288a28488ca34c8adba6fd2e8a P 036521f784e04ecca2343b74e9fda6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3525409ceea4c7da73004ad65738ffc in term 1.
I20260925 22:54:26.372915 23109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b4c7288a28488ca34c8adba6fd2e8a" candidate_uuid: "a3525409ceea4c7da73004ad65738ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ffde459ed8a47379bfe0a0b314656a0"
I20260925 22:54:26.372984 22762 leader_election.cc:302] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 036521f784e04ecca2343b74e9fda6bf, a3525409ceea4c7da73004ad65738ffc; no voters:
I20260925 22:54:26.373035 23109 raft_consensus.cc:3037] T 93b4c7288a28488ca34c8adba6fd2e8a P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:26.374146 23109 raft_consensus.cc:2454] T 93b4c7288a28488ca34c8adba6fd2e8a P 3ffde459ed8a47379bfe0a0b314656a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3525409ceea4c7da73004ad65738ffc in term 1.
I20260925 22:54:26.374451 23184 raft_consensus.cc:2781] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:26.374662 23184 raft_consensus.cc:686] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc [term 1 LEADER]: Becoming Leader. State: Replica: a3525409ceea4c7da73004ad65738ffc, State: Running, Role: LEADER
I20260925 22:54:26.374922 23184 consensus_queue.cc:237] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:26.377264 22671 catalog_manager.cc:5495] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc reported cstate change: term changed from 0 to 1, leader changed from <none> to a3525409ceea4c7da73004ad65738ffc (127.20.114.193). New cstate: current_term: 1 leader_uuid: "a3525409ceea4c7da73004ad65738ffc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } health_report { overall_health: UNKNOWN } } }
I20260925 22:54:26.378139 22658 catalog_manager.cc:4841] TS 036521f784e04ecca2343b74e9fda6bf (127.20.114.194:39451): tablet bf411b41bade4ad39a32e05cca404163 (table test-table [id=567ccfedd9134ca0bea6b956dae8692c]) successfully deleted
I20260925 22:54:26.399956 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:26.395525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57394 (local address 127.20.114.193:44777)
0925 22:54:26.395640 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:26.395645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:26.395660 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:54:26.395774 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:26.395777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:26.395840 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:26.395956 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:26.396193 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.397046 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.398224 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.398609 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.398702 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:26.399001 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:26.399566 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:26.399676 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:26.399752 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 22:54:26.399810 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:54:26.412436 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:26.408320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:38253 (local address 127.20.114.194:39451)
0925 22:54:26.408455 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:26.408461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:26.408475 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:54:26.408524 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:26.408527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:26.408578 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:26.408662 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:26.408952 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.409786 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.411104 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.411504 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.411616 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:26.411936 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:26.412026 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:26.412100 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:26.412219 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 22:54:26.412279 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:54:26.412457 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:26.407758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:39451 (local address 127.20.114.193:38253)
0925 22:54:26.408062 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:26.408077 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:26.408168 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:26.408687 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:26.408690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:26.408698 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:54:26.408818 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:26.408824 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.409901 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.409904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:26.410978 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:26.410988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.411626 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.411629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:26.411726 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:26.411784 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:54:26.412120 (+ 336us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:54:26.412126 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:54:26.412192 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 22:54:26.412272 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":119,"threads_started":1}
I20260925 22:54:26.420126 23109 raft_consensus.cc:1261] T 93b4c7288a28488ca34c8adba6fd2e8a P 3ffde459ed8a47379bfe0a0b314656a0 [term 1 FOLLOWER]: Refusing update from remote peer a3525409ceea4c7da73004ad65738ffc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:26.420172 22969 raft_consensus.cc:1261] T 93b4c7288a28488ca34c8adba6fd2e8a P 036521f784e04ecca2343b74e9fda6bf [term 1 FOLLOWER]: Refusing update from remote peer a3525409ceea4c7da73004ad65738ffc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:26.420603 23184 consensus_queue.cc:1035] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc [LEADER]: Connected to new peer: Peer: permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:26.421051 23184 consensus_queue.cc:1035] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:26.441190 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:26.437460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:44777 (local address 127.20.114.194:40385)
0925 22:54:26.437663 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:26.437678 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:26.437757 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:26.438158 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:26.438161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:26.438168 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:54:26.438259 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:26.438265 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.439288 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.439290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:26.439878 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:26.439886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.440441 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.440445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:26.440529 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:26.440567 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:54:26.440893 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:54:26.440899 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:54:26.440965 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 22:54:26.441052 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260925 22:54:26.441180 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:26.437533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:40385 (local address 127.20.114.193:44777)
0925 22:54:26.437892 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:26.437899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:26.437911 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:54:26.437948 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:26.437951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:26.437993 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:26.438055 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:26.438363 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.439179 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.439989 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:26.440320 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:26.440435 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:26.440683 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:26.440761 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:26.440826 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:26.440998 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 22:54:26.441054 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293}
W20260925 22:54:26.441902 23209 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 22:54:28.013660 22670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:28.013806 22670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:28.016171 22670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d: Sending DeleteTablet for 3 replicas of tablet 93b4c7288a28488ca34c8adba6fd2e8a
I20260925 22:54:28.016681 22800 tablet_service.cc:1514] Processing DeleteTablet for tablet 93b4c7288a28488ca34c8adba6fd2e8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:28 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:57378
I20260925 22:54:28.016716 22949 tablet_service.cc:1514] Processing DeleteTablet for tablet 93b4c7288a28488ca34c8adba6fd2e8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:28 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:33438
I20260925 22:54:28.016839 23089 tablet_service.cc:1514] Processing DeleteTablet for tablet 93b4c7288a28488ca34c8adba6fd2e8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:28 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:54302
I20260925 22:54:28.017225 23268 tablet_replica.cc:330] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc: stopping tablet replica
I20260925 22:54:28.017390 23268 raft_consensus.cc:2229] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:54:28.017419 23269 tablet_replica.cc:330] T 93b4c7288a28488ca34c8adba6fd2e8a P 036521f784e04ecca2343b74e9fda6bf: stopping tablet replica
I20260925 22:54:28.017525 23269 raft_consensus.cc:2229] T 93b4c7288a28488ca34c8adba6fd2e8a P 036521f784e04ecca2343b74e9fda6bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:28.017697 23270 tablet_replica.cc:330] T 93b4c7288a28488ca34c8adba6fd2e8a P 3ffde459ed8a47379bfe0a0b314656a0: stopping tablet replica
I20260925 22:54:28.017711 23268 raft_consensus.cc:2258] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:28.017855 23270 raft_consensus.cc:2229] T 93b4c7288a28488ca34c8adba6fd2e8a P 3ffde459ed8a47379bfe0a0b314656a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:28.018016 23269 raft_consensus.cc:2258] T 93b4c7288a28488ca34c8adba6fd2e8a P 036521f784e04ecca2343b74e9fda6bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:28.018147 23270 raft_consensus.cc:2258] T 93b4c7288a28488ca34c8adba6fd2e8a P 3ffde459ed8a47379bfe0a0b314656a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:28.018932 23269 ts_tablet_manager.cc:1905] T 93b4c7288a28488ca34c8adba6fd2e8a P 036521f784e04ecca2343b74e9fda6bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:28.019491 23270 ts_tablet_manager.cc:1905] T 93b4c7288a28488ca34c8adba6fd2e8a P 3ffde459ed8a47379bfe0a0b314656a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:28.020067 23268 ts_tablet_manager.cc:1905] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:28.021711 23269 ts_tablet_manager.cc:1918] T 93b4c7288a28488ca34c8adba6fd2e8a P 036521f784e04ecca2343b74e9fda6bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:54:28.021795 23269 log.cc:1196] T 93b4c7288a28488ca34c8adba6fd2e8a P 036521f784e04ecca2343b74e9fda6bf: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/93b4c7288a28488ca34c8adba6fd2e8a
I20260925 22:54:28.022064 23269 ts_tablet_manager.cc:1939] T 93b4c7288a28488ca34c8adba6fd2e8a P 036521f784e04ecca2343b74e9fda6bf: Deleting consensus metadata
I20260925 22:54:28.022178 23270 ts_tablet_manager.cc:1918] T 93b4c7288a28488ca34c8adba6fd2e8a P 3ffde459ed8a47379bfe0a0b314656a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:54:28.022261 23270 log.cc:1196] T 93b4c7288a28488ca34c8adba6fd2e8a P 3ffde459ed8a47379bfe0a0b314656a0: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/93b4c7288a28488ca34c8adba6fd2e8a
I20260925 22:54:28.022528 23270 ts_tablet_manager.cc:1939] T 93b4c7288a28488ca34c8adba6fd2e8a P 3ffde459ed8a47379bfe0a0b314656a0: Deleting consensus metadata
I20260925 22:54:28.022847 23268 ts_tablet_manager.cc:1918] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:54:28.022938 23268 log.cc:1196] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-0/wal/wals/93b4c7288a28488ca34c8adba6fd2e8a
I20260925 22:54:28.022980 22658 catalog_manager.cc:4841] TS 036521f784e04ecca2343b74e9fda6bf (127.20.114.194:39451): tablet 93b4c7288a28488ca34c8adba6fd2e8a (table test-table [id=892f5394c71a4673a63e0e21dd9ca404]) successfully deleted
I20260925 22:54:28.023216 23268 ts_tablet_manager.cc:1939] T 93b4c7288a28488ca34c8adba6fd2e8a P a3525409ceea4c7da73004ad65738ffc: Deleting consensus metadata
I20260925 22:54:28.023211 22658 catalog_manager.cc:4841] TS 3ffde459ed8a47379bfe0a0b314656a0 (127.20.114.195:43441): tablet 93b4c7288a28488ca34c8adba6fd2e8a (table test-table [id=892f5394c71a4673a63e0e21dd9ca404]) successfully deleted
I20260925 22:54:28.023797 22659 catalog_manager.cc:4841] TS a3525409ceea4c7da73004ad65738ffc (127.20.114.193:44777): tablet 93b4c7288a28488ca34c8adba6fd2e8a (table test-table [id=892f5394c71a4673a63e0e21dd9ca404]) successfully deleted
Sep 25 22:54:28 dist-test-slave-x1lh krb5kdc[22627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376868, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:54:28 dist-test-slave-x1lh krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376868, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:28.045297 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:28.036828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36700 (local address 127.20.114.254:42893)
0925 22:54:28.037140 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:28.037147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:28.037161 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:54:28.037229 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:28.037232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:28.037273 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:28.037354 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:28.038114 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:28.038709 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:28.039551 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:28.039784 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:28.039870 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:28.042442 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:28.042445 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:28.042447 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:28.042470 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:28.043827 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:28.044209 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:28.044211 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:28.044213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:28.044288 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:28.044542 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:28.044544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:28.044545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:28.044713 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:28.044813 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:28.045066 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 22:54:28.045167 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":162,"threads_started":1}
I20260925 22:54:28.047999 22670 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:54:28.048497 22670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:54:28.053124 22800 tablet_service.cc:1467] Processing CreateTablet for tablet 88f9c4117283485abb2dd84fe3792e49 (DEFAULT_TABLE table=test-table [id=0b300b06b1344641b7fe7a3e00526542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:28.053180 22949 tablet_service.cc:1467] Processing CreateTablet for tablet 88f9c4117283485abb2dd84fe3792e49 (DEFAULT_TABLE table=test-table [id=0b300b06b1344641b7fe7a3e00526542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:28.053260 23089 tablet_service.cc:1467] Processing CreateTablet for tablet 88f9c4117283485abb2dd84fe3792e49 (DEFAULT_TABLE table=test-table [id=0b300b06b1344641b7fe7a3e00526542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:28.053432 22800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f9c4117283485abb2dd84fe3792e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:28.053445 22949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f9c4117283485abb2dd84fe3792e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:28.053567 23089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f9c4117283485abb2dd84fe3792e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:28.055869 23281 tablet_bootstrap.cc:492] T 88f9c4117283485abb2dd84fe3792e49 P a3525409ceea4c7da73004ad65738ffc: Bootstrap starting.
I20260925 22:54:28.055868 23280 tablet_bootstrap.cc:492] T 88f9c4117283485abb2dd84fe3792e49 P 036521f784e04ecca2343b74e9fda6bf: Bootstrap starting.
I20260925 22:54:28.055971 23279 tablet_bootstrap.cc:492] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0: Bootstrap starting.
I20260925 22:54:28.056845 23281 tablet_bootstrap.cc:654] T 88f9c4117283485abb2dd84fe3792e49 P a3525409ceea4c7da73004ad65738ffc: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:28.056845 23280 tablet_bootstrap.cc:654] T 88f9c4117283485abb2dd84fe3792e49 P 036521f784e04ecca2343b74e9fda6bf: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:28.056994 23279 tablet_bootstrap.cc:654] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:28.058079 23279 tablet_bootstrap.cc:492] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0: No bootstrap required, opened a new log
I20260925 22:54:28.058099 23281 tablet_bootstrap.cc:492] T 88f9c4117283485abb2dd84fe3792e49 P a3525409ceea4c7da73004ad65738ffc: No bootstrap required, opened a new log
I20260925 22:54:28.058198 23281 ts_tablet_manager.cc:1397] T 88f9c4117283485abb2dd84fe3792e49 P a3525409ceea4c7da73004ad65738ffc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:54:28.058197 23279 ts_tablet_manager.cc:1397] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:28.058355 23280 tablet_bootstrap.cc:492] T 88f9c4117283485abb2dd84fe3792e49 P 036521f784e04ecca2343b74e9fda6bf: No bootstrap required, opened a new log
I20260925 22:54:28.058449 23280 ts_tablet_manager.cc:1397] T 88f9c4117283485abb2dd84fe3792e49 P 036521f784e04ecca2343b74e9fda6bf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 22:54:28.058609 23279 raft_consensus.cc:348] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:28.058621 23281 raft_consensus.cc:348] T 88f9c4117283485abb2dd84fe3792e49 P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:28.058707 23279 raft_consensus.cc:374] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:28.058728 23281 raft_consensus.cc:374] T 88f9c4117283485abb2dd84fe3792e49 P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:28.058740 23279 raft_consensus.cc:729] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ffde459ed8a47379bfe0a0b314656a0, State: Initialized, Role: FOLLOWER
I20260925 22:54:28.058768 23281 raft_consensus.cc:729] T 88f9c4117283485abb2dd84fe3792e49 P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3525409ceea4c7da73004ad65738ffc, State: Initialized, Role: FOLLOWER
I20260925 22:54:28.058843 23280 raft_consensus.cc:348] T 88f9c4117283485abb2dd84fe3792e49 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:28.058861 23279 consensus_queue.cc:260] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:28.058935 23280 raft_consensus.cc:374] T 88f9c4117283485abb2dd84fe3792e49 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:28.058884 23281 consensus_queue.cc:260] T 88f9c4117283485abb2dd84fe3792e49 P a3525409ceea4c7da73004ad65738ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:28.058966 23280 raft_consensus.cc:729] T 88f9c4117283485abb2dd84fe3792e49 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036521f784e04ecca2343b74e9fda6bf, State: Initialized, Role: FOLLOWER
I20260925 22:54:28.059180 23281 ts_tablet_manager.cc:1428] T 88f9c4117283485abb2dd84fe3792e49 P a3525409ceea4c7da73004ad65738ffc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:28.059212 23279 ts_tablet_manager.cc:1428] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:28.059556 23280 consensus_queue.cc:260] T 88f9c4117283485abb2dd84fe3792e49 P 036521f784e04ecca2343b74e9fda6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:28.059868 23280 ts_tablet_manager.cc:1428] T 88f9c4117283485abb2dd84fe3792e49 P 036521f784e04ecca2343b74e9fda6bf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:54:28.106413 23285 raft_consensus.cc:482] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:54:28.106606 23285 raft_consensus.cc:504] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:28.107116 23285 leader_election.cc:288] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3525409ceea4c7da73004ad65738ffc (127.20.114.193:44777), 036521f784e04ecca2343b74e9fda6bf (127.20.114.194:39451)
I20260925 22:54:28.107462 22829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f9c4117283485abb2dd84fe3792e49" candidate_uuid: "3ffde459ed8a47379bfe0a0b314656a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3525409ceea4c7da73004ad65738ffc" is_pre_election: true
I20260925 22:54:28.107529 22969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f9c4117283485abb2dd84fe3792e49" candidate_uuid: "3ffde459ed8a47379bfe0a0b314656a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "036521f784e04ecca2343b74e9fda6bf" is_pre_election: true
I20260925 22:54:28.107610 22829 raft_consensus.cc:2454] T 88f9c4117283485abb2dd84fe3792e49 P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ffde459ed8a47379bfe0a0b314656a0 in term 0.
I20260925 22:54:28.107681 22969 raft_consensus.cc:2454] T 88f9c4117283485abb2dd84fe3792e49 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ffde459ed8a47379bfe0a0b314656a0 in term 0.
I20260925 22:54:28.107945 23043 leader_election.cc:302] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ffde459ed8a47379bfe0a0b314656a0, a3525409ceea4c7da73004ad65738ffc; no voters:
I20260925 22:54:28.108114 23285 raft_consensus.cc:2781] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:54:28.108187 23285 raft_consensus.cc:482] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:54:28.108234 23285 raft_consensus.cc:3037] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:28.109323 23285 raft_consensus.cc:504] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:28.109766 23285 leader_election.cc:288] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0 [CANDIDATE]: Term 1 election: Requested vote from peers a3525409ceea4c7da73004ad65738ffc (127.20.114.193:44777), 036521f784e04ecca2343b74e9fda6bf (127.20.114.194:39451)
I20260925 22:54:28.110105 22829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f9c4117283485abb2dd84fe3792e49" candidate_uuid: "3ffde459ed8a47379bfe0a0b314656a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3525409ceea4c7da73004ad65738ffc"
I20260925 22:54:28.110234 22829 raft_consensus.cc:3037] T 88f9c4117283485abb2dd84fe3792e49 P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:28.110214 22969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f9c4117283485abb2dd84fe3792e49" candidate_uuid: "3ffde459ed8a47379bfe0a0b314656a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "036521f784e04ecca2343b74e9fda6bf"
I20260925 22:54:28.110401 22969 raft_consensus.cc:3037] T 88f9c4117283485abb2dd84fe3792e49 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:28.111351 22969 raft_consensus.cc:2454] T 88f9c4117283485abb2dd84fe3792e49 P 036521f784e04ecca2343b74e9fda6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ffde459ed8a47379bfe0a0b314656a0 in term 1.
I20260925 22:54:28.111438 22829 raft_consensus.cc:2454] T 88f9c4117283485abb2dd84fe3792e49 P a3525409ceea4c7da73004ad65738ffc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ffde459ed8a47379bfe0a0b314656a0 in term 1.
I20260925 22:54:28.111690 23042 leader_election.cc:302] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 036521f784e04ecca2343b74e9fda6bf, 3ffde459ed8a47379bfe0a0b314656a0; no voters:
I20260925 22:54:28.111855 23285 raft_consensus.cc:2781] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:28.112059 23285 raft_consensus.cc:686] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0 [term 1 LEADER]: Becoming Leader. State: Replica: 3ffde459ed8a47379bfe0a0b314656a0, State: Running, Role: LEADER
I20260925 22:54:28.112210 23285 consensus_queue.cc:237] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:28.113828 22669 catalog_manager.cc:5495] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ffde459ed8a47379bfe0a0b314656a0 (127.20.114.195). New cstate: current_term: 1 leader_uuid: "3ffde459ed8a47379bfe0a0b314656a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } health_report { overall_health: HEALTHY } } }
I20260925 22:54:28.130039 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:28.126775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.20.114.195:43441)
0925 22:54:28.127027 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:28.127037 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:28.127052 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:54:28.127091 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:28.127093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:28.127141 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:28.127235 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:28.127506 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:28.128047 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:28.128945 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:28.129180 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:28.129291 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:28.129565 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:28.129643 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:28.129721 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:28.129818 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 22:54:28.129868 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
I20260925 22:54:28.136451 22969 raft_consensus.cc:1261] T 88f9c4117283485abb2dd84fe3792e49 P 036521f784e04ecca2343b74e9fda6bf [term 1 FOLLOWER]: Refusing update from remote peer 3ffde459ed8a47379bfe0a0b314656a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:28.136478 22829 raft_consensus.cc:1261] T 88f9c4117283485abb2dd84fe3792e49 P a3525409ceea4c7da73004ad65738ffc [term 1 FOLLOWER]: Refusing update from remote peer 3ffde459ed8a47379bfe0a0b314656a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:28.136837 23285 consensus_queue.cc:1035] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:28.137034 23286 consensus_queue.cc:1035] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:28.139762 23250 mvcc.cc:205] Tried to move back new op lower bound from 7333383651883196416 to 7333383651788787712. Current Snapshot: MvccSnapshot[applied={T|T < 7333383651883196416}]
I20260925 22:54:28.166913 22671 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:28.167068 22671 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:28.168952 22671 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d: Sending DeleteTablet for 3 replicas of tablet 88f9c4117283485abb2dd84fe3792e49
I20260925 22:54:28.169452 22800 tablet_service.cc:1514] Processing DeleteTablet for tablet 88f9c4117283485abb2dd84fe3792e49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:28 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:57378
I20260925 22:54:28.169579 23089 tablet_service.cc:1514] Processing DeleteTablet for tablet 88f9c4117283485abb2dd84fe3792e49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:28 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:54302
I20260925 22:54:28.169677 23268 tablet_replica.cc:330] T 88f9c4117283485abb2dd84fe3792e49 P a3525409ceea4c7da73004ad65738ffc: stopping tablet replica
I20260925 22:54:28.169725 23270 tablet_replica.cc:330] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0: stopping tablet replica
I20260925 22:54:28.169775 22949 tablet_service.cc:1514] Processing DeleteTablet for tablet 88f9c4117283485abb2dd84fe3792e49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:28 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:33438
I20260925 22:54:28.169824 23270 raft_consensus.cc:2229] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:54:28.169816 23268 raft_consensus.cc:2229] T 88f9c4117283485abb2dd84fe3792e49 P a3525409ceea4c7da73004ad65738ffc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:28.169926 23269 tablet_replica.cc:330] T 88f9c4117283485abb2dd84fe3792e49 P 036521f784e04ecca2343b74e9fda6bf: stopping tablet replica
I20260925 22:54:28.169950 23268 raft_consensus.cc:2258] T 88f9c4117283485abb2dd84fe3792e49 P a3525409ceea4c7da73004ad65738ffc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:28.169996 23270 raft_consensus.cc:2258] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:28.170027 23269 raft_consensus.cc:2229] T 88f9c4117283485abb2dd84fe3792e49 P 036521f784e04ecca2343b74e9fda6bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:28.170122 23269 raft_consensus.cc:2258] T 88f9c4117283485abb2dd84fe3792e49 P 036521f784e04ecca2343b74e9fda6bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:28.170264 22671 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:54:28.170652 22671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:54:28.170739 23270 ts_tablet_manager.cc:1905] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:28.170745 23268 ts_tablet_manager.cc:1905] T 88f9c4117283485abb2dd84fe3792e49 P a3525409ceea4c7da73004ad65738ffc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:28.170976 23269 ts_tablet_manager.cc:1905] T 88f9c4117283485abb2dd84fe3792e49 P 036521f784e04ecca2343b74e9fda6bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:28.173014 23268 ts_tablet_manager.cc:1918] T 88f9c4117283485abb2dd84fe3792e49 P a3525409ceea4c7da73004ad65738ffc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:54:28.173066 23268 log.cc:1196] T 88f9c4117283485abb2dd84fe3792e49 P a3525409ceea4c7da73004ad65738ffc: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-0/wal/wals/88f9c4117283485abb2dd84fe3792e49
I20260925 22:54:28.173269 23268 ts_tablet_manager.cc:1939] T 88f9c4117283485abb2dd84fe3792e49 P a3525409ceea4c7da73004ad65738ffc: Deleting consensus metadata
I20260925 22:54:28.173322 23270 ts_tablet_manager.cc:1918] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:54:28.173322 23269 ts_tablet_manager.cc:1918] T 88f9c4117283485abb2dd84fe3792e49 P 036521f784e04ecca2343b74e9fda6bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:54:28.173388 23270 log.cc:1196] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/88f9c4117283485abb2dd84fe3792e49
I20260925 22:54:28.173400 23269 log.cc:1196] T 88f9c4117283485abb2dd84fe3792e49 P 036521f784e04ecca2343b74e9fda6bf: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/88f9c4117283485abb2dd84fe3792e49
I20260925 22:54:28.173656 23270 ts_tablet_manager.cc:1939] T 88f9c4117283485abb2dd84fe3792e49 P 3ffde459ed8a47379bfe0a0b314656a0: Deleting consensus metadata
I20260925 22:54:28.173656 23269 ts_tablet_manager.cc:1939] T 88f9c4117283485abb2dd84fe3792e49 P 036521f784e04ecca2343b74e9fda6bf: Deleting consensus metadata
I20260925 22:54:28.173782 22659 catalog_manager.cc:4841] TS a3525409ceea4c7da73004ad65738ffc (127.20.114.193:44777): tablet 88f9c4117283485abb2dd84fe3792e49 (table test-table [id=0b300b06b1344641b7fe7a3e00526542]) successfully deleted
I20260925 22:54:28.174407 22658 catalog_manager.cc:4841] TS 036521f784e04ecca2343b74e9fda6bf (127.20.114.194:39451): tablet 88f9c4117283485abb2dd84fe3792e49 (table test-table [id=0b300b06b1344641b7fe7a3e00526542]) successfully deleted
I20260925 22:54:28.174733 22658 catalog_manager.cc:4841] TS 3ffde459ed8a47379bfe0a0b314656a0 (127.20.114.195:43441): tablet 88f9c4117283485abb2dd84fe3792e49 (table test-table [id=0b300b06b1344641b7fe7a3e00526542]) successfully deleted
I20260925 22:54:28.177205 22800 tablet_service.cc:1467] Processing CreateTablet for tablet 68a777191f534ea3add97f96018b1130 (DEFAULT_TABLE table=test-table [id=dfb908febdd34903b401be931eb959d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:28.177459 22800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a777191f534ea3add97f96018b1130. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:28.179127 23281 tablet_bootstrap.cc:492] T 68a777191f534ea3add97f96018b1130 P a3525409ceea4c7da73004ad65738ffc: Bootstrap starting.
I20260925 22:54:28.179275 22949 tablet_service.cc:1467] Processing CreateTablet for tablet 68a777191f534ea3add97f96018b1130 (DEFAULT_TABLE table=test-table [id=dfb908febdd34903b401be931eb959d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:28.179534 22949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a777191f534ea3add97f96018b1130. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:28.180003 23281 tablet_bootstrap.cc:654] T 68a777191f534ea3add97f96018b1130 P a3525409ceea4c7da73004ad65738ffc: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:28.181094 23281 tablet_bootstrap.cc:492] T 68a777191f534ea3add97f96018b1130 P a3525409ceea4c7da73004ad65738ffc: No bootstrap required, opened a new log
I20260925 22:54:28.181164 23281 ts_tablet_manager.cc:1397] T 68a777191f534ea3add97f96018b1130 P a3525409ceea4c7da73004ad65738ffc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:28.181546 23281 raft_consensus.cc:348] T 68a777191f534ea3add97f96018b1130 P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:28.181620 23280 tablet_bootstrap.cc:492] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf: Bootstrap starting.
I20260925 22:54:28.181672 23281 raft_consensus.cc:374] T 68a777191f534ea3add97f96018b1130 P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:28.181710 23281 raft_consensus.cc:729] T 68a777191f534ea3add97f96018b1130 P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3525409ceea4c7da73004ad65738ffc, State: Initialized, Role: FOLLOWER
I20260925 22:54:28.181854 23281 consensus_queue.cc:260] T 68a777191f534ea3add97f96018b1130 P a3525409ceea4c7da73004ad65738ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:28.182124 23281 ts_tablet_manager.cc:1428] T 68a777191f534ea3add97f96018b1130 P a3525409ceea4c7da73004ad65738ffc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:54:28.182199 23089 tablet_service.cc:1467] Processing CreateTablet for tablet 68a777191f534ea3add97f96018b1130 (DEFAULT_TABLE table=test-table [id=dfb908febdd34903b401be931eb959d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:28.182420 23089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a777191f534ea3add97f96018b1130. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:28.182461 23280 tablet_bootstrap.cc:654] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:28.183490 23280 tablet_bootstrap.cc:492] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf: No bootstrap required, opened a new log
I20260925 22:54:28.183597 23280 ts_tablet_manager.cc:1397] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 22:54:28.184026 23280 raft_consensus.cc:348] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:28.184141 23280 raft_consensus.cc:374] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:28.184180 23280 raft_consensus.cc:729] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036521f784e04ecca2343b74e9fda6bf, State: Initialized, Role: FOLLOWER
I20260925 22:54:28.184315 23280 consensus_queue.cc:260] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:28.184572 23280 ts_tablet_manager.cc:1428] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:54:28.184948 23279 tablet_bootstrap.cc:492] T 68a777191f534ea3add97f96018b1130 P 3ffde459ed8a47379bfe0a0b314656a0: Bootstrap starting.
I20260925 22:54:28.185793 23279 tablet_bootstrap.cc:654] T 68a777191f534ea3add97f96018b1130 P 3ffde459ed8a47379bfe0a0b314656a0: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:28.186743 23279 tablet_bootstrap.cc:492] T 68a777191f534ea3add97f96018b1130 P 3ffde459ed8a47379bfe0a0b314656a0: No bootstrap required, opened a new log
I20260925 22:54:28.186846 23279 ts_tablet_manager.cc:1397] T 68a777191f534ea3add97f96018b1130 P 3ffde459ed8a47379bfe0a0b314656a0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:54:28.187247 23279 raft_consensus.cc:348] T 68a777191f534ea3add97f96018b1130 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:28.187356 23279 raft_consensus.cc:374] T 68a777191f534ea3add97f96018b1130 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:28.187407 23279 raft_consensus.cc:729] T 68a777191f534ea3add97f96018b1130 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ffde459ed8a47379bfe0a0b314656a0, State: Initialized, Role: FOLLOWER
I20260925 22:54:28.187536 23279 consensus_queue.cc:260] T 68a777191f534ea3add97f96018b1130 P 3ffde459ed8a47379bfe0a0b314656a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:28.187829 23279 ts_tablet_manager.cc:1428] T 68a777191f534ea3add97f96018b1130 P 3ffde459ed8a47379bfe0a0b314656a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:28.311957 23254 raft_consensus.cc:482] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:54:28.312141 23254 raft_consensus.cc:504] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:28.312601 23254 leader_election.cc:288] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3525409ceea4c7da73004ad65738ffc (127.20.114.193:44777), 3ffde459ed8a47379bfe0a0b314656a0 (127.20.114.195:43441)
I20260925 22:54:28.313122 23109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a777191f534ea3add97f96018b1130" candidate_uuid: "036521f784e04ecca2343b74e9fda6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ffde459ed8a47379bfe0a0b314656a0" is_pre_election: true
I20260925 22:54:28.313123 22829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a777191f534ea3add97f96018b1130" candidate_uuid: "036521f784e04ecca2343b74e9fda6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3525409ceea4c7da73004ad65738ffc" is_pre_election: true
I20260925 22:54:28.313283 22829 raft_consensus.cc:2454] T 68a777191f534ea3add97f96018b1130 P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 036521f784e04ecca2343b74e9fda6bf in term 0.
I20260925 22:54:28.313284 23109 raft_consensus.cc:2454] T 68a777191f534ea3add97f96018b1130 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 036521f784e04ecca2343b74e9fda6bf in term 0.
I20260925 22:54:28.313719 22903 leader_election.cc:302] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 036521f784e04ecca2343b74e9fda6bf, a3525409ceea4c7da73004ad65738ffc; no voters:
I20260925 22:54:28.313916 23254 raft_consensus.cc:2781] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:54:28.313985 23254 raft_consensus.cc:482] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:54:28.314029 23254 raft_consensus.cc:3037] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:28.315039 23254 raft_consensus.cc:504] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:28.315399 23254 leader_election.cc:288] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf [CANDIDATE]: Term 1 election: Requested vote from peers a3525409ceea4c7da73004ad65738ffc (127.20.114.193:44777), 3ffde459ed8a47379bfe0a0b314656a0 (127.20.114.195:43441)
I20260925 22:54:28.315788 22829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a777191f534ea3add97f96018b1130" candidate_uuid: "036521f784e04ecca2343b74e9fda6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3525409ceea4c7da73004ad65738ffc"
I20260925 22:54:28.315869 23109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a777191f534ea3add97f96018b1130" candidate_uuid: "036521f784e04ecca2343b74e9fda6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ffde459ed8a47379bfe0a0b314656a0"
I20260925 22:54:28.315974 22829 raft_consensus.cc:3037] T 68a777191f534ea3add97f96018b1130 P a3525409ceea4c7da73004ad65738ffc [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:28.316041 23109 raft_consensus.cc:3037] T 68a777191f534ea3add97f96018b1130 P 3ffde459ed8a47379bfe0a0b314656a0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:28.316921 22829 raft_consensus.cc:2454] T 68a777191f534ea3add97f96018b1130 P a3525409ceea4c7da73004ad65738ffc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 036521f784e04ecca2343b74e9fda6bf in term 1.
I20260925 22:54:28.317098 23109 raft_consensus.cc:2454] T 68a777191f534ea3add97f96018b1130 P 3ffde459ed8a47379bfe0a0b314656a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 036521f784e04ecca2343b74e9fda6bf in term 1.
I20260925 22:54:28.317215 22903 leader_election.cc:302] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 036521f784e04ecca2343b74e9fda6bf, a3525409ceea4c7da73004ad65738ffc; no voters:
I20260925 22:54:28.317394 23254 raft_consensus.cc:2781] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:28.317538 23254 raft_consensus.cc:686] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf [term 1 LEADER]: Becoming Leader. State: Replica: 036521f784e04ecca2343b74e9fda6bf, State: Running, Role: LEADER
I20260925 22:54:28.317687 23254 consensus_queue.cc:237] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } }
I20260925 22:54:28.319402 22669 catalog_manager.cc:5495] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 036521f784e04ecca2343b74e9fda6bf (127.20.114.194). New cstate: current_term: 1 leader_uuid: "036521f784e04ecca2343b74e9fda6bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "036521f784e04ecca2343b74e9fda6bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 39451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 } health_report { overall_health: UNKNOWN } } }
I20260925 22:54:28.344166 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:28.341038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33446 (local address 127.20.114.194:39451)
0925 22:54:28.341341 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:28.341348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:28.341359 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:54:28.341395 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:28.341397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:28.341441 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:28.341515 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:28.341738 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:28.342255 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:28.343095 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:28.343307 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:28.343381 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:28.343673 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:28.343741 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:28.343795 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:28.343920 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 22:54:28.343964 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":147,"threads_started":1}
I20260925 22:54:28.354378 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:28.351074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:54305 (local address 127.20.114.194:39451)
0925 22:54:28.351190 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:28.351195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:28.351318 (+ 123us) server_negotiation.cc:407] Connection header received
0925 22:54:28.351474 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:28.351477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:28.351523 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:28.351602 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:28.351862 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:28.352417 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:28.353337 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:28.353560 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:28.353669 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:28.353894 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:28.353971 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:28.354056 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:28.354164 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 22:54:28.354216 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":65}
I20260925 22:54:28.354378 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:28.350905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:39451 (local address 127.20.114.194:54305)
0925 22:54:28.351193 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:28.351210 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:28.351375 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:28.351624 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:28.351626 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:28.351632 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:54:28.351765 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:28.351770 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:28.352548 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:28.352552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:28.353226 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:28.353234 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:28.353669 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:28.353672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:28.353760 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:28.353780 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:54:28.354059 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:54:28.354066 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 22:54:28.354155 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 22:54:28.354240 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":22,"thread_start_us":104,"threads_started":1}
I20260925 22:54:28.360661 23109 raft_consensus.cc:1261] T 68a777191f534ea3add97f96018b1130 P 3ffde459ed8a47379bfe0a0b314656a0 [term 1 FOLLOWER]: Refusing update from remote peer 036521f784e04ecca2343b74e9fda6bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:28.360807 22829 raft_consensus.cc:1261] T 68a777191f534ea3add97f96018b1130 P a3525409ceea4c7da73004ad65738ffc [term 1 FOLLOWER]: Refusing update from remote peer 036521f784e04ecca2343b74e9fda6bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:28.361174 23254 consensus_queue.cc:1035] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3525409ceea4c7da73004ad65738ffc" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:28.361407 23246 consensus_queue.cc:1035] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ffde459ed8a47379bfe0a0b314656a0" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 22:54:28.375262 23247 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 22:54:29.929137 22671 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:29.929279 22671 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:29.931669 22671 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9351c311fc446649b41e5bedd96812d: Sending DeleteTablet for 3 replicas of tablet 68a777191f534ea3add97f96018b1130
I20260925 22:54:29.932210 22949 tablet_service.cc:1514] Processing DeleteTablet for tablet 68a777191f534ea3add97f96018b1130 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:29 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:33438
I20260925 22:54:29.932308 23089 tablet_service.cc:1514] Processing DeleteTablet for tablet 68a777191f534ea3add97f96018b1130 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:29 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:54302
I20260925 22:54:29.932339 22800 tablet_service.cc:1514] Processing DeleteTablet for tablet 68a777191f534ea3add97f96018b1130 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:29 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:57378
I20260925 22:54:29.932525 23324 tablet_replica.cc:330] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf: stopping tablet replica
I20260925 22:54:29.932559 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 22744
I20260925 22:54:29.932657 23324 raft_consensus.cc:2229] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:54:29.933893 23324 raft_consensus.cc:2258] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:29.937551 23326 tablet_replica.cc:330] T 68a777191f534ea3add97f96018b1130 P 3ffde459ed8a47379bfe0a0b314656a0: stopping tablet replica
I20260925 22:54:29.937717 23326 raft_consensus.cc:2229] T 68a777191f534ea3add97f96018b1130 P 3ffde459ed8a47379bfe0a0b314656a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:29.937841 23326 raft_consensus.cc:2258] T 68a777191f534ea3add97f96018b1130 P 3ffde459ed8a47379bfe0a0b314656a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:29.937848 23324 ts_tablet_manager.cc:1905] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:29.938558 23326 ts_tablet_manager.cc:1905] T 68a777191f534ea3add97f96018b1130 P 3ffde459ed8a47379bfe0a0b314656a0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 22:54:29.940261 22659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.114.193:44777 (error 108)
I20260925 22:54:29.940476 23324 ts_tablet_manager.cc:1918] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:54:29.940572 23324 log.cc:1196] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/68a777191f534ea3add97f96018b1130
I20260925 22:54:29.940918 23324 ts_tablet_manager.cc:1939] T 68a777191f534ea3add97f96018b1130 P 036521f784e04ecca2343b74e9fda6bf: Deleting consensus metadata
I20260925 22:54:29.941121 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 22884
I20260925 22:54:29.942507 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:29.942022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:44777 (local address 127.0.0.1:57406)
0925 22:54:29.942344 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:29.942411 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.193:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":90,"threads_started":1}
W20260925 22:54:29.942600 22659 catalog_manager.cc:4584] TS a3525409ceea4c7da73004ad65738ffc (127.20.114.193:44777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 68a777191f534ea3add97f96018b1130: Network error: Client connection negotiation failed: client connection to 127.20.114.193:44777: connect: Connection refused (error 111)
I20260925 22:54:29.943013 23326 ts_tablet_manager.cc:1918] T 68a777191f534ea3add97f96018b1130 P 3ffde459ed8a47379bfe0a0b314656a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:54:29.943070 23326 log.cc:1196] T 68a777191f534ea3add97f96018b1130 P 3ffde459ed8a47379bfe0a0b314656a0: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/68a777191f534ea3add97f96018b1130
I20260925 22:54:29.943255 23326 ts_tablet_manager.cc:1939] T 68a777191f534ea3add97f96018b1130 P 3ffde459ed8a47379bfe0a0b314656a0: Deleting consensus metadata
I20260925 22:54:29.943976 22658 catalog_manager.cc:4841] TS 3ffde459ed8a47379bfe0a0b314656a0 (127.20.114.195:43441): tablet 68a777191f534ea3add97f96018b1130 (table test-table [id=dfb908febdd34903b401be931eb959d9]) successfully deleted
I20260925 22:54:29.947762 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 23024
I20260925 22:54:29.948494 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:29.947750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:39451 (local address 127.0.0.1:33462)
0925 22:54:29.948340 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:29.948401 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.194:39451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 22:54:29.954483 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 22640
2026-09-25T22:54:29Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6292 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:54:30 dist-test-slave-x1lh krb5kdc[23331](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:54:30 dist-test-slave-x1lh krb5kdc[23331](info): set up 2 sockets
Sep 25 22:54:30 dist-test-slave-x1lh krb5kdc[23331](info): commencing operation
krb5kdc: starting...
W20260925 22:54:30.988704 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:54:31 dist-test-slave-x1lh krb5kdc[23331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:54:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:54:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:54:31.110149 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:37697
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:38093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:37697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:31.218521 23344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:31.218816 23344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:31.218894 23344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:31.222844 23344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:31.222956 23344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:31.222990 23344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:31.223037 23344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:31.223067 23344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:31.228258 23344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:38093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:37697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:37697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:31.229403 23344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:31.230451 23344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:31.239142 23352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:31.239184 23349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:31.239197 23350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:31.239296 23344 server_base.cc:1034] running on GCE node
I20260925 22:54:31.240080 23344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:31.241214 23344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:31.242401 23344 hybrid_clock.cc:648] HybridClock initialized: now 1790376871242366 us; error 46 us; skew 500 ppm
Sep 25 22:54:31 dist-test-slave-x1lh krb5kdc[23331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376871, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:31.245190 23344 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:54:31.246047 23344 webserver.cc:460] Webserver started at http://127.20.114.254:39591/ using document root <none> and password file <none>
I20260925 22:54:31.246619 23344 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:31.246709 23344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:31.246928 23344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:31.249071 23344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "7b56e954bd11439b81178b7ceeb0f2db"
format_stamp: "Formatted at 2026-09-25 22:54:31 on dist-test-slave-x1lh"
server_key: "3dfa115039ae6700232ce805ec9d6538"
server_key_iv: "c50dda6df2088c02be37cdecf25a608e"
server_key_version: "encryptionkey@0"
I20260925 22:54:31.249660 23344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "7b56e954bd11439b81178b7ceeb0f2db"
format_stamp: "Formatted at 2026-09-25 22:54:31 on dist-test-slave-x1lh"
server_key: "3dfa115039ae6700232ce805ec9d6538"
server_key_iv: "c50dda6df2088c02be37cdecf25a608e"
server_key_version: "encryptionkey@0"
I20260925 22:54:31.253394 23344 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:54:31.255716 23359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:31.257004 23344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:31.257145 23344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "7b56e954bd11439b81178b7ceeb0f2db"
format_stamp: "Formatted at 2026-09-25 22:54:31 on dist-test-slave-x1lh"
server_key: "3dfa115039ae6700232ce805ec9d6538"
server_key_iv: "c50dda6df2088c02be37cdecf25a608e"
server_key_version: "encryptionkey@0"
I20260925 22:54:31.257267 23344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:31.279422 23344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:31.282447 23344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:31.282662 23344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:31.289758 23344 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:37697
I20260925 22:54:31.289783 23411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:37697 every 8 connection(s)
I20260925 22:54:31.290706 23344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:54:31.293474 23412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:31.295955 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 23344
I20260925 22:54:31.296051 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:54:31.296309 20939 external_mini_cluster.cc:1428] Setting key 17d03b7a13844d2a0906c22fc6b74f12
I20260925 22:54:31.299742 23412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b56e954bd11439b81178b7ceeb0f2db: Bootstrap starting.
I20260925 22:54:31.302383 23412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b56e954bd11439b81178b7ceeb0f2db: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:31.303198 23412 log.cc:824] T 00000000000000000000000000000000 P 7b56e954bd11439b81178b7ceeb0f2db: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:31.305217 23412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b56e954bd11439b81178b7ceeb0f2db: No bootstrap required, opened a new log
Sep 25 22:54:31 dist-test-slave-x1lh krb5kdc[23331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:31.308760 23412 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b56e954bd11439b81178b7ceeb0f2db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b56e954bd11439b81178b7ceeb0f2db" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37697 } }
I20260925 22:54:31.309084 23412 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b56e954bd11439b81178b7ceeb0f2db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:31.309149 23412 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b56e954bd11439b81178b7ceeb0f2db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b56e954bd11439b81178b7ceeb0f2db, State: Initialized, Role: FOLLOWER
I20260925 22:54:31.309638 23412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b56e954bd11439b81178b7ceeb0f2db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b56e954bd11439b81178b7ceeb0f2db" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37697 } }
I20260925 22:54:31.309799 23412 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b56e954bd11439b81178b7ceeb0f2db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:54:31.309887 23412 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b56e954bd11439b81178b7ceeb0f2db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:54:31.310011 23412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b56e954bd11439b81178b7ceeb0f2db [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:31.310886 23412 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b56e954bd11439b81178b7ceeb0f2db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b56e954bd11439b81178b7ceeb0f2db" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37697 } }
I20260925 22:54:31.311223 23412 leader_election.cc:302] T 00000000000000000000000000000000 P 7b56e954bd11439b81178b7ceeb0f2db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b56e954bd11439b81178b7ceeb0f2db; no voters:
I20260925 22:54:31.311520 23412 leader_election.cc:288] T 00000000000000000000000000000000 P 7b56e954bd11439b81178b7ceeb0f2db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:54:31.311616 23417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b56e954bd11439b81178b7ceeb0f2db [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:31.311786 23417 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b56e954bd11439b81178b7ceeb0f2db [term 1 LEADER]: Becoming Leader. State: Replica: 7b56e954bd11439b81178b7ceeb0f2db, State: Running, Role: LEADER
I20260925 22:54:31.312078 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:31.297806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.20.114.254:37697)
0925 22:54:31.298230 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:31.298248 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:31.298292 (+ 44us) server_negotiation.cc:407] Connection header received
0925 22:54:31.299105 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:31.299128 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:31.299530 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:31.299906 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:31.300584 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:31.302107 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:31.302979 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:31.303667 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:31.303788 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:31.306568 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:31.306573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:31.306585 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:31.306609 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:31.308763 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:31.309290 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:31.309301 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:31.309305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:31.309425 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:31.309732 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:31.309735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:31.309737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:31.310256 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:31.310466 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:31.310755 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 22:54:31.311088 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":112,"threads_started":1}
I20260925 22:54:31.312101 23417 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b56e954bd11439b81178b7ceeb0f2db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b56e954bd11439b81178b7ceeb0f2db" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37697 } }
I20260925 22:54:31.312793 23412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b56e954bd11439b81178b7ceeb0f2db [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:31.313278 23419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b56e954bd11439b81178b7ceeb0f2db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b56e954bd11439b81178b7ceeb0f2db. Latest consensus state: current_term: 1 leader_uuid: "7b56e954bd11439b81178b7ceeb0f2db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b56e954bd11439b81178b7ceeb0f2db" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37697 } } }
I20260925 22:54:31.313395 23419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b56e954bd11439b81178b7ceeb0f2db [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:31.314092 23418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b56e954bd11439b81178b7ceeb0f2db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b56e954bd11439b81178b7ceeb0f2db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b56e954bd11439b81178b7ceeb0f2db" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37697 } } }
I20260925 22:54:31.314189 23418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b56e954bd11439b81178b7ceeb0f2db [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:31.314648 23421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:54:31.317824 23421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:54:31.323323 23421 catalog_manager.cc:1347] Generated new cluster ID: f36fb25980fe43c094dc4f1e6ded35ce
I20260925 22:54:31.323385 23421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:31.332267 23421 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:54:31.333266 23421 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:31.342201 23421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b56e954bd11439b81178b7ceeb0f2db: Generated new TSK 0
I20260925 22:54:31.342973 23421 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.114.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
I20260925 22:54:31.381445 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37697
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:38093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:31.491564 23438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:31.491837 23438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:31.491882 23438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:31.496011 23438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:31.496074 23438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:31.496162 23438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:54:31.501441 23438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:38093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:31.502735 23438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:31.504016 23438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:31.513273 23443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:31.513285 23446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:31.513285 23444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:31.513350 23438 server_base.cc:1034] running on GCE node
I20260925 22:54:31.513934 23438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:31.514569 23438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:31.515723 23438 hybrid_clock.cc:648] HybridClock initialized: now 1790376871515692 us; error 43 us; skew 500 ppm
Sep 25 22:54:31 dist-test-slave-x1lh krb5kdc[23331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376871, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:31.518519 23438 init.cc:377] Logged in from keytab as kudu/127.20.114.193@KRBTEST.COM (short username kudu)
I20260925 22:54:31.519379 23438 webserver.cc:460] Webserver started at http://127.20.114.193:36223/ using document root <none> and password file <none>
I20260925 22:54:31.519945 23438 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:31.520006 23438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:31.520217 23438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:31.522326 23438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "7f175372579945e5b5d745e2026503c2"
format_stamp: "Formatted at 2026-09-25 22:54:31 on dist-test-slave-x1lh"
server_key: "8a8e175793eeee3e3004aaba0cecc1c4"
server_key_iv: "b22d32e0a966d4f8d9ba5576cff26da6"
server_key_version: "encryptionkey@0"
I20260925 22:54:31.522891 23438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "7f175372579945e5b5d745e2026503c2"
format_stamp: "Formatted at 2026-09-25 22:54:31 on dist-test-slave-x1lh"
server_key: "8a8e175793eeee3e3004aaba0cecc1c4"
server_key_iv: "b22d32e0a966d4f8d9ba5576cff26da6"
server_key_version: "encryptionkey@0"
I20260925 22:54:31.526688 23438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:54:31.529248 23453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:31.530480 23438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:54:31.530611 23438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "7f175372579945e5b5d745e2026503c2"
format_stamp: "Formatted at 2026-09-25 22:54:31 on dist-test-slave-x1lh"
server_key: "8a8e175793eeee3e3004aaba0cecc1c4"
server_key_iv: "b22d32e0a966d4f8d9ba5576cff26da6"
server_key_version: "encryptionkey@0"
I20260925 22:54:31.530720 23438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:31.567545 23438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:31.571058 23438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:31.571277 23438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:31.571913 23438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:31.572757 23438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:31.572804 23438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:31.572845 23438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:31.572922 23438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:31.581787 23438 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:42343
I20260925 22:54:31.581879 23566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:42343 every 8 connection(s)
I20260925 22:54:31.582718 23438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:54:31.588090 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 23438
I20260925 22:54:31.588179 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:54:31.588506 20939 external_mini_cluster.cc:1428] Setting key a0a43d7db9c4c4141a2e809026c6ebee
Sep 25 22:54:31 dist-test-slave-x1lh krb5kdc[23331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376871, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:31.597126 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:31.584485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:44097 (local address 127.20.114.254:37697)
0925 22:54:31.584631 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:31.584636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:31.585533 (+ 897us) server_negotiation.cc:407] Connection header received
0925 22:54:31.586750 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:31.586753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:31.586798 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:31.586872 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:31.588002 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:31.588670 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:31.590005 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:31.590243 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:31.590355 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:31.593111 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:31.593116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:31.593121 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:31.593151 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:31.594769 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:31.595430 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:31.595433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:31.595434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:31.595510 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:31.595923 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:31.595925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:31.595926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:31.596091 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:31.596162 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:31.596823 (+ 661us) server_negotiation.cc:299] Negotiation successful
0925 22:54:31.597006 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:54:31.598237 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:31.584964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37697 (local address 127.20.114.193:44097)
0925 22:54:31.585382 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:31.585424 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:31.586556 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:31.587033 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:31.587039 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:31.587392 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:31.587846 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:31.587858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:31.588790 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:31.588794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:31.589878 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:31.589888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:31.590388 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:31.590392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:31.590562 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:31.591222 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:31.591248 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:31.592948 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:31.594903 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:31.594912 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:31.594930 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:31.595303 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:31.595608 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:31.595614 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:31.595617 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:31.595815 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:31.596197 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:31.596205 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:31.596553 (+ 348us) client_negotiation.cc:769] Sending connection context
0925 22:54:31.596769 (+ 216us) client_negotiation.cc:240] Negotiation successful
0925 22:54:31.597136 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.20.114.194@KRBTEST.COM; defaulting to no policy
I20260925 22:54:31.599536 23567 heartbeater.cc:347] Connected to a master server at 127.20.114.254:37697
I20260925 22:54:31.599823 23567 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:31.600363 23567 heartbeater.cc:511] Master 127.20.114.254:37697 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.194@KRBTEST.COM" created.
I20260925 22:54:31.602036 23376 ts_manager.cc:194] Registered new tserver with Master: 7f175372579945e5b5d745e2026503c2 (127.20.114.193:42343)
I20260925 22:54:31.603601 23376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:44097
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
I20260925 22:54:31.616624 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37697
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:38093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:31.725816 23572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:31.726078 23572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:31.726153 23572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:31.730033 23572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:31.730109 23572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:31.730190 23572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:54:31.735397 23572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:38093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:31.736366 23572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:31.737351 23572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:31.746585 23580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:31.746599 23578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:31.746670 23572 server_base.cc:1034] running on GCE node
W20260925 22:54:31.746599 23577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:31.747206 23572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:31.747805 23572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:31.748987 23572 hybrid_clock.cc:648] HybridClock initialized: now 1790376871748961 us; error 44 us; skew 500 ppm
Sep 25 22:54:31 dist-test-slave-x1lh krb5kdc[23331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376871, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:31.751577 23572 init.cc:377] Logged in from keytab as kudu/127.20.114.194@KRBTEST.COM (short username kudu)
I20260925 22:54:31.752429 23572 webserver.cc:460] Webserver started at http://127.20.114.194:33229/ using document root <none> and password file <none>
I20260925 22:54:31.753028 23572 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:31.753119 23572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:31.753328 23572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:31.755281 23572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "cb5e2aed863c424eb896a255f9c3af44"
format_stamp: "Formatted at 2026-09-25 22:54:31 on dist-test-slave-x1lh"
server_key: "a7c8ca6f2b79018bb01f5308dd2157a2"
server_key_iv: "dd3382d0793324a39fe51c454009db97"
server_key_version: "encryptionkey@0"
I20260925 22:54:31.755887 23572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "cb5e2aed863c424eb896a255f9c3af44"
format_stamp: "Formatted at 2026-09-25 22:54:31 on dist-test-slave-x1lh"
server_key: "a7c8ca6f2b79018bb01f5308dd2157a2"
server_key_iv: "dd3382d0793324a39fe51c454009db97"
server_key_version: "encryptionkey@0"
I20260925 22:54:31.759413 23572 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:54:31.761790 23587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:31.763046 23572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:31.763176 23572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "cb5e2aed863c424eb896a255f9c3af44"
format_stamp: "Formatted at 2026-09-25 22:54:31 on dist-test-slave-x1lh"
server_key: "a7c8ca6f2b79018bb01f5308dd2157a2"
server_key_iv: "dd3382d0793324a39fe51c454009db97"
server_key_version: "encryptionkey@0"
I20260925 22:54:31.763280 23572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:31.783277 23572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:31.786207 23572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:31.786405 23572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:31.786998 23572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:31.787783 23572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:31.787853 23572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:31.787930 23572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:31.787966 23572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:31.796277 23572 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:36093
I20260925 22:54:31.796342 23700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:36093 every 8 connection(s)
I20260925 22:54:31.797194 23572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:54:31.802889 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 23572
I20260925 22:54:31.803017 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:54:31.803330 20939 external_mini_cluster.cc:1428] Setting key 8de2e04501532ba19a357922f70b7d88
Sep 25 22:54:31 dist-test-slave-x1lh krb5kdc[23331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376871, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:31.812093 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:31.798979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:41921 (local address 127.20.114.254:37697)
0925 22:54:31.799118 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:31.799122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:31.799970 (+ 848us) server_negotiation.cc:407] Connection header received
0925 22:54:31.801252 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:31.801255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:31.801315 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:31.801439 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:31.802501 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:31.803215 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:31.804548 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:31.804958 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:31.805050 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:31.807777 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:31.807784 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:31.807791 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:31.807826 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:31.809761 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:31.810261 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:31.810268 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:31.810273 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:31.810380 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:31.810717 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:31.810724 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:31.810729 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:31.811026 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:31.811156 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:31.811718 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 22:54:31.811876 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:54:31.812752 23703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:31.799384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37697 (local address 127.20.114.194:41921)
0925 22:54:31.799814 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:31.799853 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:31.801057 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:31.801602 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:31.801609 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:31.801927 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:31.802369 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:31.802381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:31.803335 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:31.803341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:31.804430 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:31.804440 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:31.805104 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:31.805109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:31.805321 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:31.805973 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:31.805997 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:31.807607 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:31.809896 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:31.809900 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:31.809912 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:31.810153 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:31.810492 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:31.810495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:31.810496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:31.810621 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:31.811158 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:31.811166 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:31.811456 (+ 290us) client_negotiation.cc:769] Sending connection context
0925 22:54:31.811621 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 22:54:31.811863 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":127,"threads_started":1}
I20260925 22:54:31.814091 23701 heartbeater.cc:347] Connected to a master server at 127.20.114.254:37697
I20260925 22:54:31.814368 23701 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.20.114.195@KRBTEST.COM; defaulting to no policy
I20260925 22:54:31.815007 23701 heartbeater.cc:511] Master 127.20.114.254:37697 requested a full tablet report, sending...
I20260925 22:54:31.816126 23376 ts_manager.cc:194] Registered new tserver with Master: cb5e2aed863c424eb896a255f9c3af44 (127.20.114.194:36093)
I20260925 22:54:31.816809 23376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:41921
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
I20260925 22:54:31.834170 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37697
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:38093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:31.946893 23706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:31.947180 23706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:31.947260 23706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:31.951344 23706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:31.951437 23706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:31.951551 23706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:54:31.956902 23706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:38093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:31.957991 23706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:31.959002 23706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:31.968608 23711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:31.968544 23712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:31.968552 23714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:31.968814 23706 server_base.cc:1034] running on GCE node
I20260925 22:54:31.969286 23706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:31.969970 23706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:31.971127 23706 hybrid_clock.cc:648] HybridClock initialized: now 1790376871971110 us; error 29 us; skew 500 ppm
Sep 25 22:54:31 dist-test-slave-x1lh krb5kdc[23331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376871, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:31.974148 23706 init.cc:377] Logged in from keytab as kudu/127.20.114.195@KRBTEST.COM (short username kudu)
I20260925 22:54:31.975135 23706 webserver.cc:460] Webserver started at http://127.20.114.195:46037/ using document root <none> and password file <none>
I20260925 22:54:31.975656 23706 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:31.975710 23706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:31.975870 23706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:31.977945 23706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "cef63d9323c546b286226c0f67b59839"
format_stamp: "Formatted at 2026-09-25 22:54:31 on dist-test-slave-x1lh"
server_key: "ad8486ebcff15300dc98da995fda7406"
server_key_iv: "34a5224d5e933af356bef359ec6e5b67"
server_key_version: "encryptionkey@0"
I20260925 22:54:31.978438 23706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "cef63d9323c546b286226c0f67b59839"
format_stamp: "Formatted at 2026-09-25 22:54:31 on dist-test-slave-x1lh"
server_key: "ad8486ebcff15300dc98da995fda7406"
server_key_iv: "34a5224d5e933af356bef359ec6e5b67"
server_key_version: "encryptionkey@0"
I20260925 22:54:31.982009 23706 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 22:54:31.984338 23721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:31.985635 23706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:54:31.985728 23706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "cef63d9323c546b286226c0f67b59839"
format_stamp: "Formatted at 2026-09-25 22:54:31 on dist-test-slave-x1lh"
server_key: "ad8486ebcff15300dc98da995fda7406"
server_key_iv: "34a5224d5e933af356bef359ec6e5b67"
server_key_version: "encryptionkey@0"
I20260925 22:54:31.985810 23706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:31.995128 23706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:31.997958 23706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:31.998111 23706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:31.998629 23706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:31.999403 23706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:31.999456 23706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:31.999496 23706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:31.999518 23706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:32.008126 23706 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:46387
I20260925 22:54:32.008148 23834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:46387 every 8 connection(s)
I20260925 22:54:32.009115 23706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:54:32.010298 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 23706
I20260925 22:54:32.010411 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:54:32.010704 20939 external_mini_cluster.cc:1428] Setting key 87aeacc1e5db792af6b2f0b375f05e2c
Sep 25 22:54:32 dist-test-slave-x1lh krb5kdc[23331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376871, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:32.022970 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:32.011017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:60183 (local address 127.20.114.254:37697)
0925 22:54:32.011142 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:32.011146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:32.012103 (+ 957us) server_negotiation.cc:407] Connection header received
0925 22:54:32.013708 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:32.013712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:32.013770 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:32.013869 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:32.015074 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:32.015605 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:32.016815 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:32.017197 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:32.017307 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:32.019826 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:32.019829 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:32.019831 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:32.019857 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:32.021242 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:32.021696 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:32.021698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:32.021699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:32.021768 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:32.022052 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:32.022054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:32.022055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:32.022218 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:32.022291 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:32.022755 (+ 464us) server_negotiation.cc:299] Negotiation successful
0925 22:54:32.022848 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 22:54:32.024148 23837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:32.011450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37697 (local address 127.20.114.195:60183)
0925 22:54:32.011969 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:32.012022 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:32.013386 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:32.014063 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:32.014070 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:32.014412 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:32.014939 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:32.014953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:32.015758 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:32.015761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:32.016679 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:32.016687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:32.017354 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:32.017357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:32.017546 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:32.018171 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:32.018187 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:32.019675 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:32.021341 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:32.021346 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:32.021357 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:32.021611 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:32.021844 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:32.021846 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:32.021847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:32.021970 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:32.022300 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:32.022306 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:32.022558 (+ 252us) client_negotiation.cc:769] Sending connection context
0925 22:54:32.022713 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 22:54:32.023040 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":159,"threads_started":1}
I20260925 22:54:32.025327 23835 heartbeater.cc:347] Connected to a master server at 127.20.114.254:37697
I20260925 22:54:32.025588 23835 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:32.026085 23835 heartbeater.cc:511] Master 127.20.114.254:37697 requested a full tablet report, sending...
I20260925 22:54:32.027103 23376 ts_manager.cc:194] Registered new tserver with Master: cef63d9323c546b286226c0f67b59839 (127.20.114.195:46387)
I20260925 22:54:32.027690 23376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:60183
I20260925 22:54:32.037072 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 22:54:32.049964 23415 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:32.048323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.20.114.254:37697)
0925 22:54:32.048527 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:32.048533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:32.048672 (+ 139us) server_negotiation.cc:407] Connection header received
0925 22:54:32.048839 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:32.048843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:32.048931 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:32.049032 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:32.049804 (+ 772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55516: BlockingRecv error: recv got EOF from 127.0.0.1:55516 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
I20260925 22:54:32.051218 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 23438
I20260925 22:54:32.056627 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 23572
I20260925 22:54:32.061715 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 23706
I20260925 22:54:32.066645 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 23344
2026-09-25T22:54:32Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2110 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:54:32 dist-test-slave-x1lh krb5kdc[23847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:54:32 dist-test-slave-x1lh krb5kdc[23847](info): set up 2 sockets
Sep 25 22:54:32 dist-test-slave-x1lh krb5kdc[23847](info): commencing operation
krb5kdc: starting...
W20260925 22:54:33.100751 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:54:33 dist-test-slave-x1lh krb5kdc[23847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:54:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:54:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:54:33.240556 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:46481
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:46481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:33.350558 23860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:33.350844 23860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:33.350888 23860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:33.354921 23860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:33.354984 23860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:33.355003 23860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:33.355020 23860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:33.355036 23860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:33.360437 23860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:46481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:46481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:33.362409 23860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:33.364147 23860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:33.374209 23865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:33.374202 23868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:33.374313 23860 server_base.cc:1034] running on GCE node
W20260925 22:54:33.374226 23866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:33.375120 23860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:33.376332 23860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:33.377514 23860 hybrid_clock.cc:648] HybridClock initialized: now 1790376873377496 us; error 33 us; skew 500 ppm
Sep 25 22:54:33 dist-test-slave-x1lh krb5kdc[23847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376873, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:33.381106 23860 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:54:33.382045 23860 webserver.cc:460] Webserver started at http://127.20.114.254:34297/ using document root <none> and password file <none>
I20260925 22:54:33.382668 23860 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:33.382735 23860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:33.382992 23860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:33.385324 23860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "a7f617a7b47f4d79ab74b21a59c59a08"
format_stamp: "Formatted at 2026-09-25 22:54:33 on dist-test-slave-x1lh"
server_key: "c54a6a4bfbb0b241f753e9321a036d97"
server_key_iv: "75d2e6eeb332146511e3b1dcedacabfa"
server_key_version: "encryptionkey@0"
I20260925 22:54:33.385954 23860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "a7f617a7b47f4d79ab74b21a59c59a08"
format_stamp: "Formatted at 2026-09-25 22:54:33 on dist-test-slave-x1lh"
server_key: "c54a6a4bfbb0b241f753e9321a036d97"
server_key_iv: "75d2e6eeb332146511e3b1dcedacabfa"
server_key_version: "encryptionkey@0"
I20260925 22:54:33.390009 23860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260925 22:54:33.392558 23875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:33.393949 23860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:33.394107 23860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "a7f617a7b47f4d79ab74b21a59c59a08"
format_stamp: "Formatted at 2026-09-25 22:54:33 on dist-test-slave-x1lh"
server_key: "c54a6a4bfbb0b241f753e9321a036d97"
server_key_iv: "75d2e6eeb332146511e3b1dcedacabfa"
server_key_version: "encryptionkey@0"
I20260925 22:54:33.394232 23860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:33.407856 23860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:33.411140 23860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:33.411368 23860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:33.418751 23860 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:46481
I20260925 22:54:33.419030 23927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:46481 every 8 connection(s)
I20260925 22:54:33.419802 23860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:54:33.422688 23928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:33.426994 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 23860
I20260925 22:54:33.427155 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:54:33.427433 20939 external_mini_cluster.cc:1428] Setting key ef604061d19a986bdd79c318302947bd
I20260925 22:54:33.429109 23928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08: Bootstrap starting.
I20260925 22:54:33.431599 23928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:33.432307 23928 log.cc:824] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:33.434111 23928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08: No bootstrap required, opened a new log
Sep 25 22:54:33 dist-test-slave-x1lh krb5kdc[23847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:33.437793 23928 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f617a7b47f4d79ab74b21a59c59a08" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 46481 } }
I20260925 22:54:33.438050 23928 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:33.438151 23928 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7f617a7b47f4d79ab74b21a59c59a08, State: Initialized, Role: FOLLOWER
I20260925 22:54:33.438673 23928 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f617a7b47f4d79ab74b21a59c59a08" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 46481 } }
I20260925 22:54:33.438802 23928 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:54:33.438885 23928 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:54:33.438982 23928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:33.439838 23928 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f617a7b47f4d79ab74b21a59c59a08" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 46481 } }
I20260925 22:54:33.440228 23928 leader_election.cc:302] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7f617a7b47f4d79ab74b21a59c59a08; no voters:
I20260925 22:54:33.440560 23928 leader_election.cc:288] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:54:33.440723 23933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:33.441097 23933 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08 [term 1 LEADER]: Becoming Leader. State: Replica: a7f617a7b47f4d79ab74b21a59c59a08, State: Running, Role: LEADER
I20260925 22:54:33.441501 23933 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f617a7b47f4d79ab74b21a59c59a08" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 46481 } }
I20260925 22:54:33.441906 23928 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:33.443014 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:33.428754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52094 (local address 127.20.114.254:46481)
0925 22:54:33.429238 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:33.429259 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:33.429311 (+ 52us) server_negotiation.cc:407] Connection header received
0925 22:54:33.429983 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:33.430001 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:33.430351 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:33.430646 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:33.431529 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:33.432709 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:33.433702 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:33.434251 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:33.434441 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:33.437148 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:33.437154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:33.437168 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:33.437196 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:33.439180 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:33.439670 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:33.439677 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:33.439683 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:33.439816 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:33.440127 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:33.440131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:33.440132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:33.440670 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:33.440894 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:33.441618 (+ 724us) server_negotiation.cc:299] Negotiation successful
0925 22:54:33.441943 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":199,"threads_started":1}
I20260925 22:54:33.443288 23935 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7f617a7b47f4d79ab74b21a59c59a08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f617a7b47f4d79ab74b21a59c59a08" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 46481 } } }
I20260925 22:54:33.443413 23935 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:33.443663 23935 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7f617a7b47f4d79ab74b21a59c59a08. Latest consensus state: current_term: 1 leader_uuid: "a7f617a7b47f4d79ab74b21a59c59a08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f617a7b47f4d79ab74b21a59c59a08" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 46481 } } }
I20260925 22:54:33.443737 23935 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:33.446110 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 22:54:33.447285 23948 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:54:33.447351 23948 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:54:33.447454 23943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:54:33.448405 23943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:54:33.453785 23943 catalog_manager.cc:1347] Generated new cluster ID: 24f10c8d4c04450e965c2ee0200c9e75
I20260925 22:54:33.453841 23943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:33.462077 23943 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:54:33.462970 23943 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:33.466961 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:33.460624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52108 (local address 127.20.114.254:46481)
0925 22:54:33.460750 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:33.460755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:33.461089 (+ 334us) server_negotiation.cc:407] Connection header received
0925 22:54:33.461206 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:33.461210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:33.461268 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:33.461364 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:33.462174 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:33.463106 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:33.464192 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:33.464645 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:33.464755 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:33.465741 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:33.465745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:33.465748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:54:33.466364 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:33.466401 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:54:33.466426 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:33.466533 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:33.466717 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 22:54:33.466801 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":46}
I20260925 22:54:33.474424 23943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7f617a7b47f4d79ab74b21a59c59a08: Generated new TSK 0
I20260925 22:54:33.475421 23943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:54:33.483045 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 23860
2026-09-25T22:54:33Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1414 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:54:33 dist-test-slave-x1lh krb5kdc[23961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:54:33 dist-test-slave-x1lh krb5kdc[23961](info): set up 2 sockets
Sep 25 22:54:33 dist-test-slave-x1lh krb5kdc[23961](info): commencing operation
krb5kdc: starting...
W20260925 22:54:34.540783 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:54:34 dist-test-slave-x1lh krb5kdc[23961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:54:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:54:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:54:34.661528 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:34315
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:34315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:34.769198 23974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:34.769538 23974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:34.769649 23974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:34.773985 23974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:34.774075 23974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:34.774114 23974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:34.774147 23974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:34.774217 23974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:34.779819 23974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:34315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:34315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:34.780967 23974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:34.782001 23974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:34.790307 23982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:34.790295 23980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:34.790310 23979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:34.790565 23974 server_base.cc:1034] running on GCE node
I20260925 22:54:34.791050 23974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:34.792097 23974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:34.793249 23974 hybrid_clock.cc:648] HybridClock initialized: now 1790376874793219 us; error 41 us; skew 500 ppm
Sep 25 22:54:34 dist-test-slave-x1lh krb5kdc[23961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376874, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:34.795958 23974 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:54:34.796780 23974 webserver.cc:460] Webserver started at http://127.20.114.254:40759/ using document root <none> and password file <none>
I20260925 22:54:34.797344 23974 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:34.797405 23974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:34.797562 23974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:34.799453 23974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "5b7e4d9c53f641e0a49caa97ba019381"
format_stamp: "Formatted at 2026-09-25 22:54:34 on dist-test-slave-x1lh"
server_key: "7fa7deb7614657ee8b81186e4b194f66"
server_key_iv: "78e4584bb847d168c7048e4db2906aaf"
server_key_version: "encryptionkey@0"
I20260925 22:54:34.799952 23974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "5b7e4d9c53f641e0a49caa97ba019381"
format_stamp: "Formatted at 2026-09-25 22:54:34 on dist-test-slave-x1lh"
server_key: "7fa7deb7614657ee8b81186e4b194f66"
server_key_iv: "78e4584bb847d168c7048e4db2906aaf"
server_key_version: "encryptionkey@0"
I20260925 22:54:34.803480 23974 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 22:54:34.805785 23989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:34.807188 23974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:34.807343 23974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "5b7e4d9c53f641e0a49caa97ba019381"
format_stamp: "Formatted at 2026-09-25 22:54:34 on dist-test-slave-x1lh"
server_key: "7fa7deb7614657ee8b81186e4b194f66"
server_key_iv: "78e4584bb847d168c7048e4db2906aaf"
server_key_version: "encryptionkey@0"
I20260925 22:54:34.807461 23974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:34.822659 23974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:34.826509 23974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:34.826742 23974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:34.834404 23974 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:34315
I20260925 22:54:34.834697 24041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:34315 every 8 connection(s)
I20260925 22:54:34.835536 23974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:54:34.837320 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 23974
I20260925 22:54:34.837437 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:54:34.837736 20939 external_mini_cluster.cc:1428] Setting key 558df49d4b6c7dc4a1ab32446133654c
I20260925 22:54:34.838864 24042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:34.845629 24042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b7e4d9c53f641e0a49caa97ba019381: Bootstrap starting.
Sep 25 22:54:34 dist-test-slave-x1lh krb5kdc[23961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:34.847864 24042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b7e4d9c53f641e0a49caa97ba019381: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:34.848791 24042 log.cc:824] T 00000000000000000000000000000000 P 5b7e4d9c53f641e0a49caa97ba019381: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:34.851112 24042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b7e4d9c53f641e0a49caa97ba019381: No bootstrap required, opened a new log
I20260925 22:54:34.853052 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:34.839252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47912 (local address 127.20.114.254:34315)
0925 22:54:34.839670 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:34.839689 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:34.839730 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:54:34.840567 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:34.840591 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:34.841030 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:34.841407 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:34.842107 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:34.843576 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:34.844524 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:34.845240 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:34.845396 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:34.847911 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:34.847918 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:34.847935 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:34.847965 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:34.849985 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:34.850403 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:34.850408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:34.850413 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:34.850545 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:34.850854 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:34.850857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:34.850859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:34.851358 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:34.851538 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:34.851803 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 22:54:34.852110 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":113,"threads_started":1}
I20260925 22:54:34.855394 24042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b7e4d9c53f641e0a49caa97ba019381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7e4d9c53f641e0a49caa97ba019381" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34315 } }
I20260925 22:54:34.855621 24042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b7e4d9c53f641e0a49caa97ba019381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:34.855657 24042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b7e4d9c53f641e0a49caa97ba019381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e4d9c53f641e0a49caa97ba019381, State: Initialized, Role: FOLLOWER
I20260925 22:54:34.856047 24042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b7e4d9c53f641e0a49caa97ba019381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7e4d9c53f641e0a49caa97ba019381" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34315 } }
I20260925 22:54:34.856170 24042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b7e4d9c53f641e0a49caa97ba019381 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:54:34.856216 24042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b7e4d9c53f641e0a49caa97ba019381 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:54:34.856278 24042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b7e4d9c53f641e0a49caa97ba019381 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:34.857156 24042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b7e4d9c53f641e0a49caa97ba019381 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7e4d9c53f641e0a49caa97ba019381" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34315 } }
I20260925 22:54:34.857440 24042 leader_election.cc:302] T 00000000000000000000000000000000 P 5b7e4d9c53f641e0a49caa97ba019381 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b7e4d9c53f641e0a49caa97ba019381; no voters:
I20260925 22:54:34.857746 24042 leader_election.cc:288] T 00000000000000000000000000000000 P 5b7e4d9c53f641e0a49caa97ba019381 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:54:34.857861 24047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b7e4d9c53f641e0a49caa97ba019381 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:34.858132 24047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b7e4d9c53f641e0a49caa97ba019381 [term 1 LEADER]: Becoming Leader. State: Replica: 5b7e4d9c53f641e0a49caa97ba019381, State: Running, Role: LEADER
I20260925 22:54:34.858443 24047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b7e4d9c53f641e0a49caa97ba019381 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7e4d9c53f641e0a49caa97ba019381" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34315 } }
I20260925 22:54:34.858980 24042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b7e4d9c53f641e0a49caa97ba019381 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:34.860041 24048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b7e4d9c53f641e0a49caa97ba019381 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b7e4d9c53f641e0a49caa97ba019381" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7e4d9c53f641e0a49caa97ba019381" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34315 } } }
I20260925 22:54:34.860183 24048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b7e4d9c53f641e0a49caa97ba019381 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:34.860447 24049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b7e4d9c53f641e0a49caa97ba019381 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b7e4d9c53f641e0a49caa97ba019381. Latest consensus state: current_term: 1 leader_uuid: "5b7e4d9c53f641e0a49caa97ba019381" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7e4d9c53f641e0a49caa97ba019381" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34315 } } }
I20260925 22:54:34.860532 24049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b7e4d9c53f641e0a49caa97ba019381 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:34.860630 24056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:54:34.863853 24056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:54:34.869233 24056 catalog_manager.cc:1347] Generated new cluster ID: 95ab7f47186e4df39d408fd21ba86ab3
I20260925 22:54:34.869302 24056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:34.897312 24056 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:54:34.898566 24056 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:34.904251 24056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b7e4d9c53f641e0a49caa97ba019381: Generated new TSK 0
I20260925 22:54:34.905077 24056 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.114.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
I20260925 22:54:34.931675 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:34315
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:35.040076 24068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:35.040330 24068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:35.040374 24068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:35.044363 24068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:35.044420 24068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:35.044478 24068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:54:35.049731 24068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:34315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:35.050770 24068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:35.051790 24068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:35.061054 24073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:35.061064 24074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:35.061036 24076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:35.061501 24068 server_base.cc:1034] running on GCE node
I20260925 22:54:35.061892 24068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:35.062527 24068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:35.063711 24068 hybrid_clock.cc:648] HybridClock initialized: now 1790376875063695 us; error 29 us; skew 500 ppm
Sep 25 22:54:35 dist-test-slave-x1lh krb5kdc[23961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376875, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:35.069336 24068 init.cc:377] Logged in from keytab as kudu/127.20.114.193@KRBTEST.COM (short username kudu)
I20260925 22:54:35.070186 24068 webserver.cc:460] Webserver started at http://127.20.114.193:34285/ using document root <none> and password file <none>
I20260925 22:54:35.070744 24068 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:35.070834 24068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:35.071065 24068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:35.073143 24068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "aae9504c357645ddac0ea604e6adf34d"
format_stamp: "Formatted at 2026-09-25 22:54:35 on dist-test-slave-x1lh"
server_key: "3efab6a56fa96f6b4e3f81c95a2a57ed"
server_key_iv: "4511a1354b5f367c53ede77dded35573"
server_key_version: "encryptionkey@0"
I20260925 22:54:35.073714 24068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "aae9504c357645ddac0ea604e6adf34d"
format_stamp: "Formatted at 2026-09-25 22:54:35 on dist-test-slave-x1lh"
server_key: "3efab6a56fa96f6b4e3f81c95a2a57ed"
server_key_iv: "4511a1354b5f367c53ede77dded35573"
server_key_version: "encryptionkey@0"
I20260925 22:54:35.077374 24068 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:54:35.079903 24083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:35.081140 24068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:35.081262 24068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "aae9504c357645ddac0ea604e6adf34d"
format_stamp: "Formatted at 2026-09-25 22:54:35 on dist-test-slave-x1lh"
server_key: "3efab6a56fa96f6b4e3f81c95a2a57ed"
server_key_iv: "4511a1354b5f367c53ede77dded35573"
server_key_version: "encryptionkey@0"
I20260925 22:54:35.081367 24068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:35.108279 24068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:35.111654 24068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:35.111801 24068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:35.112370 24068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:35.113247 24068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:35.113318 24068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:35.113389 24068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:35.113440 24068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:35.122097 24068 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:38239
I20260925 22:54:35.122156 24196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:38239 every 8 connection(s)
I20260925 22:54:35.123005 24068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:54:35.127987 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 24068
I20260925 22:54:35.128085 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:54:35.128367 20939 external_mini_cluster.cc:1428] Setting key 14d09c8f458345416415abe370007dc7
Sep 25 22:54:35 dist-test-slave-x1lh krb5kdc[23961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376875, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:35.138399 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:35.124802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:53457 (local address 127.20.114.254:34315)
0925 22:54:35.124952 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:35.124956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:35.125810 (+ 854us) server_negotiation.cc:407] Connection header received
0925 22:54:35.127101 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:35.127104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:35.127158 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:35.127238 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:35.128446 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:35.129208 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:35.130899 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:35.131121 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:35.131200 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:35.134259 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:35.134265 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:35.134270 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:35.134299 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:35.136226 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:35.136723 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:35.136726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:35.136728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:35.136822 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:35.137175 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:35.137180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:35.137184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:35.137438 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:35.137544 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:35.138010 (+ 466us) server_negotiation.cc:299] Negotiation successful
0925 22:54:35.138151 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for kudu/127.20.114.194@KRBTEST.COM; defaulting to no policy
I20260925 22:54:35.139140 24199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:35.125213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:34315 (local address 127.20.114.193:53457)
0925 22:54:35.125654 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:35.125695 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:35.126881 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:35.127397 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:35.127404 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:35.127752 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:35.128297 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:35.128311 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:35.129334 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:35.129337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:35.130769 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:35.130784 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:35.131282 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:35.131288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:35.131531 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:35.132218 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:35.132240 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:35.134075 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:35.136352 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:35.136357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:35.136368 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:35.136614 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:35.136953 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:35.136956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:35.136957 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:35.137082 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:35.137546 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:35.137551 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:35.137769 (+ 218us) client_negotiation.cc:769] Sending connection context
0925 22:54:35.137938 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 22:54:35.138177 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":116,"threads_started":1}
I20260925 22:54:35.140350 24197 heartbeater.cc:347] Connected to a master server at 127.20.114.254:34315
I20260925 22:54:35.140625 24197 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:35.141165 24197 heartbeater.cc:511] Master 127.20.114.254:34315 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.194@KRBTEST.COM" created.
I20260925 22:54:35.142688 24006 ts_manager.cc:194] Registered new tserver with Master: aae9504c357645ddac0ea604e6adf34d (127.20.114.193:38239)
I20260925 22:54:35.144277 24006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:53457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
I20260925 22:54:35.157632 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:34315
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:35.266470 24202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:35.266731 24202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:35.266773 24202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:35.270659 24202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:35.270711 24202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:35.270768 24202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:54:35.275861 24202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:34315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:35.276840 24202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:35.277817 24202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:35.287632 24208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:35.287638 24210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:35.287817 24202 server_base.cc:1034] running on GCE node
W20260925 22:54:35.287645 24207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:35.288195 24202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:35.288800 24202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:35.289964 24202 hybrid_clock.cc:648] HybridClock initialized: now 1790376875289924 us; error 51 us; skew 500 ppm
Sep 25 22:54:35 dist-test-slave-x1lh krb5kdc[23961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376875, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:35.292797 24202 init.cc:377] Logged in from keytab as kudu/127.20.114.194@KRBTEST.COM (short username kudu)
I20260925 22:54:35.293713 24202 webserver.cc:460] Webserver started at http://127.20.114.194:36381/ using document root <none> and password file <none>
I20260925 22:54:35.294261 24202 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:35.294351 24202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:35.294574 24202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:35.296499 24202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "98c98c0b0d12472eacfc64fc10449966"
format_stamp: "Formatted at 2026-09-25 22:54:35 on dist-test-slave-x1lh"
server_key: "8a4b8b0be1b73fffc09032a87dab2c71"
server_key_iv: "e47142830b31e71eb4021c94e9db0715"
server_key_version: "encryptionkey@0"
I20260925 22:54:35.297112 24202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "98c98c0b0d12472eacfc64fc10449966"
format_stamp: "Formatted at 2026-09-25 22:54:35 on dist-test-slave-x1lh"
server_key: "8a4b8b0be1b73fffc09032a87dab2c71"
server_key_iv: "e47142830b31e71eb4021c94e9db0715"
server_key_version: "encryptionkey@0"
I20260925 22:54:35.300604 24202 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:54:35.302903 24217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:35.304168 24202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:35.304294 24202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "98c98c0b0d12472eacfc64fc10449966"
format_stamp: "Formatted at 2026-09-25 22:54:35 on dist-test-slave-x1lh"
server_key: "8a4b8b0be1b73fffc09032a87dab2c71"
server_key_iv: "e47142830b31e71eb4021c94e9db0715"
server_key_version: "encryptionkey@0"
I20260925 22:54:35.304402 24202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:35.323927 24202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:35.327271 24202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:35.327421 24202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:35.328044 24202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:35.328830 24202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:35.328925 24202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:35.329002 24202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:35.329037 24202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:35.337805 24202 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:35341
I20260925 22:54:35.337874 24330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:35341 every 8 connection(s)
I20260925 22:54:35.338718 24202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:54:35.344084 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 24202
I20260925 22:54:35.344223 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:54:35.344494 20939 external_mini_cluster.cc:1428] Setting key a061a121cb9d15d5eaba18825781065b
Sep 25 22:54:35 dist-test-slave-x1lh krb5kdc[23961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376875, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:35.353055 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:35.340481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:36099 (local address 127.20.114.254:34315)
0925 22:54:35.340613 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:35.340617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:35.341448 (+ 831us) server_negotiation.cc:407] Connection header received
0925 22:54:35.342797 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:35.342800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:35.342859 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:35.342940 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:35.344244 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:35.344856 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:35.346138 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:35.346371 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:35.346481 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:35.349240 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:35.349243 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:35.349245 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:35.349268 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:35.350817 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:35.351342 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:35.351346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:35.351348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:35.351435 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:35.351785 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:35.351788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:35.351790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:35.351998 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:35.352105 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:35.352768 (+ 663us) server_negotiation.cc:299] Negotiation successful
0925 22:54:35.352923 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.20.114.195@KRBTEST.COM; defaulting to no policy
I20260925 22:54:35.354342 24333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:35.340810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:34315 (local address 127.20.114.194:36099)
0925 22:54:35.341301 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:35.341342 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:35.342585 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:35.343106 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:35.343114 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:35.343460 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:35.344055 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:35.344077 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:35.345062 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:35.345066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:35.346008 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:35.346017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:35.346517 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:35.346521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:35.346701 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:35.347354 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:35.347381 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:35.349063 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:35.350950 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:35.350956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:35.350972 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:35.351235 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:35.351535 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:35.351538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:35.351541 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:35.351682 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:35.352141 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:35.352147 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:35.352494 (+ 347us) client_negotiation.cc:769] Sending connection context
0925 22:54:35.352736 (+ 242us) client_negotiation.cc:240] Negotiation successful
0925 22:54:35.353156 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":115,"threads_started":1}
I20260925 22:54:35.355758 24331 heartbeater.cc:347] Connected to a master server at 127.20.114.254:34315
I20260925 22:54:35.355976 24331 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.195@KRBTEST.COM" created.
I20260925 22:54:35.356608 24331 heartbeater.cc:511] Master 127.20.114.254:34315 requested a full tablet report, sending...
I20260925 22:54:35.357756 24006 ts_manager.cc:194] Registered new tserver with Master: 98c98c0b0d12472eacfc64fc10449966 (127.20.114.194:35341)
I20260925 22:54:35.358345 24006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:36099
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
I20260925 22:54:35.373692 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:34315
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:35.486047 24336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:35.486387 24336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:35.486474 24336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:35.490597 24336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:35.490692 24336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:35.490787 24336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:54:35.496178 24336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:34315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:35.497347 24336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:35.498610 24336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:35.508649 24342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:35.508683 24344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:35.508662 24341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:35.508642 24336 server_base.cc:1034] running on GCE node
I20260925 22:54:35.509277 24336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:35.509939 24336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:35.511098 24336 hybrid_clock.cc:648] HybridClock initialized: now 1790376875511078 us; error 30 us; skew 500 ppm
Sep 25 22:54:35 dist-test-slave-x1lh krb5kdc[23961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376875, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:35.514002 24336 init.cc:377] Logged in from keytab as kudu/127.20.114.195@KRBTEST.COM (short username kudu)
I20260925 22:54:35.514868 24336 webserver.cc:460] Webserver started at http://127.20.114.195:40825/ using document root <none> and password file <none>
I20260925 22:54:35.515436 24336 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:35.515527 24336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:35.515760 24336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:35.517856 24336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "3fbef59e38f047d7be3faab3ab9718fc"
format_stamp: "Formatted at 2026-09-25 22:54:35 on dist-test-slave-x1lh"
server_key: "3fac316d050d09976b87e2f907508561"
server_key_iv: "027ba9d475c6af5a06fe3f18b960829d"
server_key_version: "encryptionkey@0"
I20260925 22:54:35.518447 24336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "3fbef59e38f047d7be3faab3ab9718fc"
format_stamp: "Formatted at 2026-09-25 22:54:35 on dist-test-slave-x1lh"
server_key: "3fac316d050d09976b87e2f907508561"
server_key_iv: "027ba9d475c6af5a06fe3f18b960829d"
server_key_version: "encryptionkey@0"
I20260925 22:54:35.522177 24336 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:54:35.524698 24351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:35.526044 24336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:35.526193 24336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "3fbef59e38f047d7be3faab3ab9718fc"
format_stamp: "Formatted at 2026-09-25 22:54:35 on dist-test-slave-x1lh"
server_key: "3fac316d050d09976b87e2f907508561"
server_key_iv: "027ba9d475c6af5a06fe3f18b960829d"
server_key_version: "encryptionkey@0"
I20260925 22:54:35.526309 24336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:35.556511 24336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:35.560201 24336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:35.560411 24336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:35.561086 24336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:35.561923 24336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:35.561992 24336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:35.562064 24336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:35.562108 24336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:35.570796 24336 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:36811
I20260925 22:54:35.570859 24464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:36811 every 8 connection(s)
I20260925 22:54:35.571713 24336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:54:35.580312 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 24336
I20260925 22:54:35.580443 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:54:35.580778 20939 external_mini_cluster.cc:1428] Setting key 15861b472f2723bd41adc8d32d7aaf4b
Sep 25 22:54:35 dist-test-slave-x1lh krb5kdc[23961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376875, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:35.585538 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:35.573549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:41135 (local address 127.20.114.254:34315)
0925 22:54:35.573690 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:35.573694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:35.574550 (+ 856us) server_negotiation.cc:407] Connection header received
0925 22:54:35.575892 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:35.575896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:35.575952 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:35.576055 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:35.577216 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:35.577742 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:35.578879 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:35.579102 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:35.579212 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:35.582140 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:35.582143 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:35.582145 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:35.582193 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:35.583553 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:35.584010 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:35.584012 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:35.584014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:35.584082 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:35.584384 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:35.584387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:35.584388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:35.584558 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:35.584637 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:35.585253 (+ 616us) server_negotiation.cc:299] Negotiation successful
0925 22:54:35.585353 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:54:35.586421 24467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:35.573981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:34315 (local address 127.20.114.195:41135)
0925 22:54:35.574407 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:35.574447 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:35.575680 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:35.576235 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:35.576242 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:35.576586 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:35.577066 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:35.577087 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:35.577863 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:35.577867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:35.578748 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:35.578757 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:35.579248 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:35.579253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:35.579446 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:35.580101 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:35.580124 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:35.581956 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:35.583663 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:35.583667 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:35.583679 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:35.583926 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:35.584159 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:35.584161 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:35.584162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:35.584301 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:35.584650 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:35.584656 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:35.585007 (+ 351us) client_negotiation.cc:769] Sending connection context
0925 22:54:35.585222 (+ 215us) client_negotiation.cc:240] Negotiation successful
0925 22:54:35.585505 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":109,"threads_started":1}
I20260925 22:54:35.587530 24465 heartbeater.cc:347] Connected to a master server at 127.20.114.254:34315
I20260925 22:54:35.587808 24465 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:35.588263 24465 heartbeater.cc:511] Master 127.20.114.254:34315 requested a full tablet report, sending...
I20260925 22:54:35.589203 24006 ts_manager.cc:194] Registered new tserver with Master: 3fbef59e38f047d7be3faab3ab9718fc (127.20.114.195:36811)
I20260925 22:54:35.589720 24006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:41135
I20260925 22:54:35.594835 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:54:35 dist-test-slave-x1lh krb5kdc[23961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376875, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 22:54:35 dist-test-slave-x1lh krb5kdc[23961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376875, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:35.620297 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:35.612010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47914 (local address 127.20.114.254:34315)
0925 22:54:35.612167 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:35.612172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:35.612311 (+ 139us) server_negotiation.cc:407] Connection header received
0925 22:54:35.612463 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:35.612465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:35.612509 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:35.612582 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:35.613406 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:35.613912 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:35.614724 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:35.614922 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:35.614995 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:35.617398 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:35.617401 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:35.617403 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:35.617425 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:35.618666 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:35.619045 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:35.619048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:35.619049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:35.619128 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:35.619449 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:35.619451 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:35.619452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:35.619662 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:35.619762 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:35.619981 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 22:54:35.620086 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 22:54:35.620747 24006 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47914
I20260925 22:54:35.621905 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 24068
I20260925 22:54:35.627249 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 24202
I20260925 22:54:35.633265 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 24336
I20260925 22:54:35.638546 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 23974
2026-09-25T22:54:35Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2156 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:54:35 dist-test-slave-x1lh krb5kdc[24477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:54:35 dist-test-slave-x1lh krb5kdc[24477](info): set up 2 sockets
Sep 25 22:54:35 dist-test-slave-x1lh krb5kdc[24477](info): commencing operation
krb5kdc: starting...
W20260925 22:54:36.677250 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:54:36 dist-test-slave-x1lh krb5kdc[24477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:54:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:54:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:54:36.800153 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:37897
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:37897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:36.907788 24490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:36.908033 24490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:36.908072 24490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:36.912089 24490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:36.912150 24490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:36.912170 24490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:36.912186 24490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:36.912202 24490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:36.917397 24490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:37897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:37897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:36.918413 24490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:36.919431 24490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:36.927883 24498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:36.927909 24495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:36.927992 24490 server_base.cc:1034] running on GCE node
W20260925 22:54:36.927901 24496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:36.928653 24490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:36.929778 24490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:36.930939 24490 hybrid_clock.cc:648] HybridClock initialized: now 1790376876930919 us; error 32 us; skew 500 ppm
Sep 25 22:54:36 dist-test-slave-x1lh krb5kdc[24477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376876, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:36.933916 24490 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:54:36.934769 24490 webserver.cc:460] Webserver started at http://127.20.114.254:36711/ using document root <none> and password file <none>
I20260925 22:54:36.935267 24490 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:36.935325 24490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:36.935482 24490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:36.937556 24490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "14e5160d42ff4f4c991ebcfdeae3eaf6"
format_stamp: "Formatted at 2026-09-25 22:54:36 on dist-test-slave-x1lh"
server_key: "a178a8f39473d8da7d0e355368281d1b"
server_key_iv: "35db7f28a638079a00e3ee7401c5023c"
server_key_version: "encryptionkey@0"
I20260925 22:54:36.938105 24490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "14e5160d42ff4f4c991ebcfdeae3eaf6"
format_stamp: "Formatted at 2026-09-25 22:54:36 on dist-test-slave-x1lh"
server_key: "a178a8f39473d8da7d0e355368281d1b"
server_key_iv: "35db7f28a638079a00e3ee7401c5023c"
server_key_version: "encryptionkey@0"
I20260925 22:54:36.941792 24490 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:54:36.944051 24505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:36.945434 24490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:36.945569 24490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "14e5160d42ff4f4c991ebcfdeae3eaf6"
format_stamp: "Formatted at 2026-09-25 22:54:36 on dist-test-slave-x1lh"
server_key: "a178a8f39473d8da7d0e355368281d1b"
server_key_iv: "35db7f28a638079a00e3ee7401c5023c"
server_key_version: "encryptionkey@0"
I20260925 22:54:36.945676 24490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:36.963361 24490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:36.966598 24490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:36.966794 24490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:36.973752 24490 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:37897
I20260925 22:54:36.973774 24557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:37897 every 8 connection(s)
I20260925 22:54:36.974723 24490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:54:36.976312 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 24490
I20260925 22:54:36.976431 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:54:36.976732 20939 external_mini_cluster.cc:1428] Setting key 8b5282d9be59f2f057241f7942023731
I20260925 22:54:36.977707 24558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:36.983206 24558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14e5160d42ff4f4c991ebcfdeae3eaf6: Bootstrap starting.
Sep 25 22:54:36 dist-test-slave-x1lh krb5kdc[24477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:36.985670 24558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14e5160d42ff4f4c991ebcfdeae3eaf6: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:36.986389 24558 log.cc:824] T 00000000000000000000000000000000 P 14e5160d42ff4f4c991ebcfdeae3eaf6: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:36.988220 24558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14e5160d42ff4f4c991ebcfdeae3eaf6: No bootstrap required, opened a new log
I20260925 22:54:36.990352 24561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:36.978132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.20.114.254:37897)
0925 22:54:36.978554 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:36.978568 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:36.978602 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:54:36.979215 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:36.979229 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:36.979536 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:36.979854 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:36.980666 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:36.981703 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:36.982541 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:36.983011 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:36.983187 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:36.985849 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:36.985854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:36.985868 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:36.985893 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:36.987600 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:36.988027 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:36.988033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:36.988046 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:36.988187 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:36.988454 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:36.988457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:36.988458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:36.988839 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:36.989077 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:36.989392 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 22:54:36.989669 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":144,"threads_started":1}
I20260925 22:54:36.991863 24558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14e5160d42ff4f4c991ebcfdeae3eaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14e5160d42ff4f4c991ebcfdeae3eaf6" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37897 } }
I20260925 22:54:36.992098 24558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14e5160d42ff4f4c991ebcfdeae3eaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:36.992154 24558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14e5160d42ff4f4c991ebcfdeae3eaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e5160d42ff4f4c991ebcfdeae3eaf6, State: Initialized, Role: FOLLOWER
I20260925 22:54:36.992605 24558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14e5160d42ff4f4c991ebcfdeae3eaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14e5160d42ff4f4c991ebcfdeae3eaf6" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37897 } }
I20260925 22:54:36.992760 24558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14e5160d42ff4f4c991ebcfdeae3eaf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:54:36.992841 24558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14e5160d42ff4f4c991ebcfdeae3eaf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:54:36.992964 24558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14e5160d42ff4f4c991ebcfdeae3eaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:36.993821 24558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14e5160d42ff4f4c991ebcfdeae3eaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14e5160d42ff4f4c991ebcfdeae3eaf6" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37897 } }
I20260925 22:54:36.994154 24558 leader_election.cc:302] T 00000000000000000000000000000000 P 14e5160d42ff4f4c991ebcfdeae3eaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14e5160d42ff4f4c991ebcfdeae3eaf6; no voters:
I20260925 22:54:36.994457 24558 leader_election.cc:288] T 00000000000000000000000000000000 P 14e5160d42ff4f4c991ebcfdeae3eaf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:54:36.994587 24563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14e5160d42ff4f4c991ebcfdeae3eaf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:36.994827 24563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14e5160d42ff4f4c991ebcfdeae3eaf6 [term 1 LEADER]: Becoming Leader. State: Replica: 14e5160d42ff4f4c991ebcfdeae3eaf6, State: Running, Role: LEADER
I20260925 22:54:36.995118 24563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14e5160d42ff4f4c991ebcfdeae3eaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14e5160d42ff4f4c991ebcfdeae3eaf6" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37897 } }
I20260925 22:54:36.995663 24558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14e5160d42ff4f4c991ebcfdeae3eaf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:36.996925 24565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14e5160d42ff4f4c991ebcfdeae3eaf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14e5160d42ff4f4c991ebcfdeae3eaf6. Latest consensus state: current_term: 1 leader_uuid: "14e5160d42ff4f4c991ebcfdeae3eaf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14e5160d42ff4f4c991ebcfdeae3eaf6" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37897 } } }
I20260925 22:54:36.997064 24565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14e5160d42ff4f4c991ebcfdeae3eaf6 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:36.997979 24572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:54:36.998075 24564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14e5160d42ff4f4c991ebcfdeae3eaf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14e5160d42ff4f4c991ebcfdeae3eaf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14e5160d42ff4f4c991ebcfdeae3eaf6" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37897 } } }
I20260925 22:54:36.998418 24564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14e5160d42ff4f4c991ebcfdeae3eaf6 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:37.000753 24572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:54:37.006026 24572 catalog_manager.cc:1347] Generated new cluster ID: 9f9af7f2af2c4f048665d033379321ec
I20260925 22:54:37.006089 24572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:37.027954 24572 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:54:37.029315 24572 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:37.040589 24572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14e5160d42ff4f4c991ebcfdeae3eaf6: Generated new TSK 0
I20260925 22:54:37.041368 24572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.114.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
I20260925 22:54:37.071923 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37897
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:37.189254 24584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:37.189545 24584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:37.189628 24584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:37.194164 24584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:37.194228 24584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:37.194341 24584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:54:37.200495 24584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:37.201694 24584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:37.202811 24584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:37.212400 24592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:37.212400 24589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:37.212426 24590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:37.212589 24584 server_base.cc:1034] running on GCE node
I20260925 22:54:37.213109 24584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:37.213722 24584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:37.214901 24584 hybrid_clock.cc:648] HybridClock initialized: now 1790376877214889 us; error 27 us; skew 500 ppm
Sep 25 22:54:37 dist-test-slave-x1lh krb5kdc[24477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376877, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:37.217648 24584 init.cc:377] Logged in from keytab as kudu/127.20.114.193@KRBTEST.COM (short username kudu)
I20260925 22:54:37.218549 24584 webserver.cc:460] Webserver started at http://127.20.114.193:33029/ using document root <none> and password file <none>
I20260925 22:54:37.219091 24584 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:37.219174 24584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:37.219400 24584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:37.221438 24584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "5542aef9f54149958884ed0fd39413eb"
format_stamp: "Formatted at 2026-09-25 22:54:37 on dist-test-slave-x1lh"
server_key: "5d4c4e3f2ddbab61068f34b10a4d84b3"
server_key_iv: "6579297be3b4c3be58ed5e6cae9c15a3"
server_key_version: "encryptionkey@0"
I20260925 22:54:37.221971 24584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "5542aef9f54149958884ed0fd39413eb"
format_stamp: "Formatted at 2026-09-25 22:54:37 on dist-test-slave-x1lh"
server_key: "5d4c4e3f2ddbab61068f34b10a4d84b3"
server_key_iv: "6579297be3b4c3be58ed5e6cae9c15a3"
server_key_version: "encryptionkey@0"
I20260925 22:54:37.225660 24584 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:54:37.228111 24599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:37.229491 24584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 22:54:37.229647 24584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "5542aef9f54149958884ed0fd39413eb"
format_stamp: "Formatted at 2026-09-25 22:54:37 on dist-test-slave-x1lh"
server_key: "5d4c4e3f2ddbab61068f34b10a4d84b3"
server_key_iv: "6579297be3b4c3be58ed5e6cae9c15a3"
server_key_version: "encryptionkey@0"
I20260925 22:54:37.229763 24584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:37.255806 24584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:37.259297 24584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:37.259500 24584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:37.260102 24584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:37.260998 24584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:37.261065 24584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:37.261132 24584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:37.261188 24584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:37.270547 24584 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:45425
I20260925 22:54:37.270568 24712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:45425 every 8 connection(s)
I20260925 22:54:37.271440 24584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:54:37.278550 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 24584
I20260925 22:54:37.278698 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:54:37.278972 20939 external_mini_cluster.cc:1428] Setting key 7766641507f1814b2ca51e9b2067ae99
Sep 25 22:54:37 dist-test-slave-x1lh krb5kdc[24477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376877, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:37.286211 24561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:37.273282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:55549 (local address 127.20.114.254:37897)
0925 22:54:37.273417 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:37.273421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:37.274245 (+ 824us) server_negotiation.cc:407] Connection header received
0925 22:54:37.275540 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:37.275543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:37.275591 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:37.275667 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:37.276789 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:37.277420 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:37.278695 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:37.278974 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:37.279090 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:37.282128 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:37.282134 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:37.282139 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:37.282172 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:37.283936 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:37.284428 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:37.284435 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:37.284440 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:37.284545 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:37.284978 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:37.284982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:37.284984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:37.285229 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:37.285334 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:37.285817 (+ 483us) server_negotiation.cc:299] Negotiation successful
0925 22:54:37.285969 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:54:37.286844 24715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:37.273663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37897 (local address 127.20.114.193:55549)
0925 22:54:37.274096 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:37.274137 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:37.275343 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:37.275826 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:37.275832 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:37.276190 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:37.276651 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:37.276664 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:37.277537 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:37.277539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:37.278563 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:37.278576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:37.279131 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:37.279135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:37.279311 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:37.279960 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:37.279976 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:37.281916 (+ 1940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:37.284046 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:37.284051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:37.284062 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:37.284319 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:37.284660 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:37.284666 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:37.284672 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:37.284831 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:37.285339 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:37.285344 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:37.285569 (+ 225us) client_negotiation.cc:769] Sending connection context
0925 22:54:37.285726 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 22:54:37.285970 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":101,"threads_started":1}
I20260925 22:54:37.288080 24713 heartbeater.cc:347] Connected to a master server at 127.20.114.254:37897
I20260925 22:54:37.288313 24713 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.20.114.194@KRBTEST.COM; defaulting to no policy
I20260925 22:54:37.288899 24713 heartbeater.cc:511] Master 127.20.114.254:37897 requested a full tablet report, sending...
I20260925 22:54:37.290364 24522 ts_manager.cc:194] Registered new tserver with Master: 5542aef9f54149958884ed0fd39413eb (127.20.114.193:45425)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.194@KRBTEST.COM" created.
I20260925 22:54:37.291662 24522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:55549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
I20260925 22:54:37.307863 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37897
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:37.419201 24718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:37.419476 24718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:37.419592 24718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:37.423800 24718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:37.423862 24718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:37.423982 24718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:54:37.429342 24718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:37.430372 24718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:37.431403 24718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:37.440917 24723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:37.440953 24718 server_base.cc:1034] running on GCE node
W20260925 22:54:37.440907 24724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:37.440907 24726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:37.441474 24718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:37.442065 24718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:37.443219 24718 hybrid_clock.cc:648] HybridClock initialized: now 1790376877443205 us; error 24 us; skew 500 ppm
Sep 25 22:54:37 dist-test-slave-x1lh krb5kdc[24477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376877, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:37.445863 24718 init.cc:377] Logged in from keytab as kudu/127.20.114.194@KRBTEST.COM (short username kudu)
I20260925 22:54:37.446725 24718 webserver.cc:460] Webserver started at http://127.20.114.194:41787/ using document root <none> and password file <none>
I20260925 22:54:37.447273 24718 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:37.447357 24718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:37.447595 24718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:37.449673 24718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "2cad28f6cc0343b78e7b1577f4f73e9e"
format_stamp: "Formatted at 2026-09-25 22:54:37 on dist-test-slave-x1lh"
server_key: "cde81f3ca7686161724207d7c90b3167"
server_key_iv: "2a00948c3c35ee672bf00b92bd61692a"
server_key_version: "encryptionkey@0"
I20260925 22:54:37.450232 24718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "2cad28f6cc0343b78e7b1577f4f73e9e"
format_stamp: "Formatted at 2026-09-25 22:54:37 on dist-test-slave-x1lh"
server_key: "cde81f3ca7686161724207d7c90b3167"
server_key_iv: "2a00948c3c35ee672bf00b92bd61692a"
server_key_version: "encryptionkey@0"
I20260925 22:54:37.453812 24718 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:54:37.456423 24733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:37.457806 24718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:54:37.457933 24718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "2cad28f6cc0343b78e7b1577f4f73e9e"
format_stamp: "Formatted at 2026-09-25 22:54:37 on dist-test-slave-x1lh"
server_key: "cde81f3ca7686161724207d7c90b3167"
server_key_iv: "2a00948c3c35ee672bf00b92bd61692a"
server_key_version: "encryptionkey@0"
I20260925 22:54:37.458035 24718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:37.472519 24718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:37.475523 24718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:37.475718 24718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:37.476277 24718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:37.477159 24718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:37.477207 24718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:37.477269 24718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:37.477301 24718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:37.485600 24718 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:45913
I20260925 22:54:37.485666 24846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:45913 every 8 connection(s)
I20260925 22:54:37.486495 24718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:54:37.493698 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 24718
I20260925 22:54:37.493825 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:54:37.494096 20939 external_mini_cluster.cc:1428] Setting key e7c235168d424b4b58682dfde3211b4d
Sep 25 22:54:37 dist-test-slave-x1lh krb5kdc[24477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376877, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:37.500661 24561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:37.488240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:46681 (local address 127.20.114.254:37897)
0925 22:54:37.488378 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:37.488382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:37.489293 (+ 911us) server_negotiation.cc:407] Connection header received
0925 22:54:37.490683 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:37.490687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:37.490743 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:37.490854 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:37.491988 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:37.492518 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:37.493882 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:37.494118 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:37.494212 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:37.497030 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:37.497032 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:37.497035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:37.497056 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:37.498677 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:37.499181 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:37.499191 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:37.499192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:37.499265 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:37.499605 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:37.499609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:37.499610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:37.499790 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:37.499898 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:37.500386 (+ 488us) server_negotiation.cc:299] Negotiation successful
0925 22:54:37.500503 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:54:37.501566 24849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:37.488631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37897 (local address 127.20.114.194:46681)
0925 22:54:37.489123 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:37.489182 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:37.490472 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:37.491024 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:37.491031 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:37.491380 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:37.491844 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:37.491856 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:37.492659 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:37.492662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:37.493734 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:37.493747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:37.494284 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:37.494288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:37.494486 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:37.495088 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:37.495106 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:37.496839 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:37.498792 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:37.498798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:37.498812 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:37.499070 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:37.499361 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:37.499364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:37.499366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:37.499504 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:37.499895 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:37.499902 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:37.500153 (+ 251us) client_negotiation.cc:769] Sending connection context
0925 22:54:37.500335 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 22:54:37.500664 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":94,"threads_started":1}
I20260925 22:54:37.502743 24847 heartbeater.cc:347] Connected to a master server at 127.20.114.254:37897
I20260925 22:54:37.503011 24847 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.20.114.195@KRBTEST.COM; defaulting to no policy
I20260925 22:54:37.503598 24847 heartbeater.cc:511] Master 127.20.114.254:37897 requested a full tablet report, sending...
I20260925 22:54:37.504647 24522 ts_manager.cc:194] Registered new tserver with Master: 2cad28f6cc0343b78e7b1577f4f73e9e (127.20.114.194:45913)
I20260925 22:54:37.505272 24522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:46681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
I20260925 22:54:37.520041 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37897
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:37.627830 24852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:37.628110 24852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:37.628197 24852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:37.632344 24852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:37.632405 24852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:37.632519 24852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:54:37.637878 24852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:37.639134 24852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:37.640180 24852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:37.650336 24858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:37.650528 24852 server_base.cc:1034] running on GCE node
W20260925 22:54:37.650332 24857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:37.650332 24860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:37.651026 24852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:37.651726 24852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:37.652937 24852 hybrid_clock.cc:648] HybridClock initialized: now 1790376877652891 us; error 48 us; skew 500 ppm
Sep 25 22:54:37 dist-test-slave-x1lh krb5kdc[24477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376877, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:37.655673 24852 init.cc:377] Logged in from keytab as kudu/127.20.114.195@KRBTEST.COM (short username kudu)
I20260925 22:54:37.656497 24852 webserver.cc:460] Webserver started at http://127.20.114.195:39117/ using document root <none> and password file <none>
I20260925 22:54:37.657124 24852 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:37.657214 24852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:37.657446 24852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:37.659502 24852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "9fe80bebce4e44faaf0cbe769fe0358c"
format_stamp: "Formatted at 2026-09-25 22:54:37 on dist-test-slave-x1lh"
server_key: "e8bee79e843da6ada99bfa080933f07d"
server_key_iv: "9c140b0c5282dc4738de2207b6358cc6"
server_key_version: "encryptionkey@0"
I20260925 22:54:37.660069 24852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "9fe80bebce4e44faaf0cbe769fe0358c"
format_stamp: "Formatted at 2026-09-25 22:54:37 on dist-test-slave-x1lh"
server_key: "e8bee79e843da6ada99bfa080933f07d"
server_key_iv: "9c140b0c5282dc4738de2207b6358cc6"
server_key_version: "encryptionkey@0"
I20260925 22:54:37.663805 24852 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 22:54:37.666203 24867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:37.667354 24852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:54:37.667487 24852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "9fe80bebce4e44faaf0cbe769fe0358c"
format_stamp: "Formatted at 2026-09-25 22:54:37 on dist-test-slave-x1lh"
server_key: "e8bee79e843da6ada99bfa080933f07d"
server_key_iv: "9c140b0c5282dc4738de2207b6358cc6"
server_key_version: "encryptionkey@0"
I20260925 22:54:37.667604 24852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:37.703447 24852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:37.707005 24852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:37.707214 24852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:37.707815 24852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:37.708642 24852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:37.708712 24852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:37.708784 24852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:37.708837 24852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:37.717705 24852 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:38947
I20260925 22:54:37.717721 24980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:38947 every 8 connection(s)
I20260925 22:54:37.718606 24852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:54:37.726485 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 24852
I20260925 22:54:37.726642 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:54:37.726951 20939 external_mini_cluster.cc:1428] Setting key c294cdb4ae178c8783b1d0222319da57
Sep 25 22:54:37 dist-test-slave-x1lh krb5kdc[24477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376877, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:37.732089 24561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:37.720374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:35191 (local address 127.20.114.254:37897)
0925 22:54:37.720540 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:37.720546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:37.721441 (+ 895us) server_negotiation.cc:407] Connection header received
0925 22:54:37.722786 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:37.722789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:37.722836 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:37.722902 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:37.724037 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:37.724558 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:37.725672 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:37.725881 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:37.725959 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:37.728623 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:37.728626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:37.728628 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:37.728650 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:37.730104 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:37.730619 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:37.730623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:37.730624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:37.730703 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:37.731039 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:37.731042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:37.731044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:37.731224 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:37.731327 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:37.731814 (+ 487us) server_negotiation.cc:299] Negotiation successful
0925 22:54:37.731927 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 22:54:37.732856 24983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:37.720805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37897 (local address 127.20.114.195:35191)
0925 22:54:37.721292 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:37.721335 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:37.722575 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:37.723078 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:37.723085 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:37.723439 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:37.723917 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:37.723931 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:37.724682 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:37.724685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:37.725569 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:37.725577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:37.726007 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:37.726010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:37.726152 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:37.726844 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:37.726860 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:37.728453 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:37.730235 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:37.730241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:37.730255 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:37.730508 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:37.730795 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:37.730799 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:37.730801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:37.730937 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:37.731335 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:37.731344 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:37.731579 (+ 235us) client_negotiation.cc:769] Sending connection context
0925 22:54:37.731761 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 22:54:37.732064 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":97,"threads_started":1}
I20260925 22:54:37.733978 24981 heartbeater.cc:347] Connected to a master server at 127.20.114.254:37897
I20260925 22:54:37.734241 24981 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:37.734740 24981 heartbeater.cc:511] Master 127.20.114.254:37897 requested a full tablet report, sending...
I20260925 22:54:37.735666 24522 ts_manager.cc:194] Registered new tserver with Master: 9fe80bebce4e44faaf0cbe769fe0358c (127.20.114.195:38947)
I20260925 22:54:37.736254 24522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:35191
I20260925 22:54:37.740546 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:54:37 dist-test-slave-x1lh krb5kdc[24477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:54:37 dist-test-slave-x1lh krb5kdc[24477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.193@KRBTEST.COM
I20260925 22:54:37.765276 24988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:37.756263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40512 (local address 127.20.114.193:45425)
0925 22:54:37.756552 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:37.756566 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:37.756591 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:54:37.756643 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:37.756647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:37.756833 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:37.757006 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:37.757638 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:37.758390 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:37.759163 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:37.759510 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:37.759626 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:37.761976 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:37.761988 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:37.762006 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:37.762034 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:37.763506 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:37.763949 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:37.763954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:37.763955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:37.764028 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:37.764314 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:37.764319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:37.764321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:37.764546 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:37.764758 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:37.764953 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 22:54:37.765110 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":98,"threads_started":1}
Sep 25 22:54:37 dist-test-slave-x1lh krb5kdc[24477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:37.776017 24561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:37.768033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.20.114.254:37897)
0925 22:54:37.768167 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:37.768174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:37.768231 (+ 57us) server_negotiation.cc:407] Connection header received
0925 22:54:37.768373 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:37.768381 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:37.768421 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:37.768504 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:37.769128 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:37.769652 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:37.770411 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:37.770632 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:37.770742 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:37.773217 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:37.773221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:37.773223 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:37.773245 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:37.774413 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:37.774814 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:37.774819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:37.774820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:37.774897 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:37.775165 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:37.775170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:37.775172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:37.775382 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:37.775522 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:37.775742 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 22:54:37.775858 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 22:54:37.776393 24522 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35370
I20260925 22:54:37.777047 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 24584
I20260925 22:54:37.782552 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 24718
I20260925 22:54:37.787885 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 24852
I20260925 22:54:37.793102 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 24490
2026-09-25T22:54:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2154 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:54:37 dist-test-slave-x1lh krb5kdc[24993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:54:37 dist-test-slave-x1lh krb5kdc[24993](info): set up 2 sockets
Sep 25 22:54:37 dist-test-slave-x1lh krb5kdc[24993](info): commencing operation
krb5kdc: starting...
W20260925 22:54:38.833015 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:54:38 dist-test-slave-x1lh krb5kdc[24993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:54:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:54:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:54:38.952654 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:40107
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:40107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:39.059899 25006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:39.060153 25006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:39.060195 25006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:39.064069 25006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:39.064119 25006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:39.064138 25006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:39.064154 25006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:39.064169 25006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:39.069227 25006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:40107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:40107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:39.070175 25006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:39.071111 25006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:39.079140 25014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:39.079174 25011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:39.079182 25012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:39.079747 25006 server_base.cc:1034] running on GCE node
I20260925 22:54:39.080228 25006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:39.081300 25006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:39.082486 25006 hybrid_clock.cc:648] HybridClock initialized: now 1790376879082473 us; error 32 us; skew 500 ppm
Sep 25 22:54:39 dist-test-slave-x1lh krb5kdc[24993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376879, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:39.085143 25006 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:54:39.085892 25006 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:39.085985 25006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:39.086202 25006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:39.088199 25006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "07eb2420aa4b47d1989ce310777b2874"
format_stamp: "Formatted at 2026-09-25 22:54:39 on dist-test-slave-x1lh"
server_key: "25b9a17503d804ecc71b9649964f1a04"
server_key_iv: "516b90bf6b63f9b7b1784d75599f1f7a"
server_key_version: "encryptionkey@0"
I20260925 22:54:39.088783 25006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "07eb2420aa4b47d1989ce310777b2874"
format_stamp: "Formatted at 2026-09-25 22:54:39 on dist-test-slave-x1lh"
server_key: "25b9a17503d804ecc71b9649964f1a04"
server_key_iv: "516b90bf6b63f9b7b1784d75599f1f7a"
server_key_version: "encryptionkey@0"
I20260925 22:54:39.092344 25006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:54:39.094758 25020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:39.095944 25006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 22:54:39.096041 25006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "07eb2420aa4b47d1989ce310777b2874"
format_stamp: "Formatted at 2026-09-25 22:54:39 on dist-test-slave-x1lh"
server_key: "25b9a17503d804ecc71b9649964f1a04"
server_key_iv: "516b90bf6b63f9b7b1784d75599f1f7a"
server_key_version: "encryptionkey@0"
I20260925 22:54:39.096125 25006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:39.105615 25006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:39.108111 25006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:39.108247 25006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:39.114993 25006 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:40107
I20260925 22:54:39.115267 25072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:40107 every 8 connection(s)
I20260925 22:54:39.115651 25006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:54:39.118122 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 25006
I20260925 22:54:39.118249 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:54:39.118438 25073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:39.118546 20939 external_mini_cluster.cc:1428] Setting key 0f938b5f29f22ec6ed31bc63bc65302e
I20260925 22:54:39.124554 25073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874: Bootstrap starting.
I20260925 22:54:39.127043 25073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874: Neither blocks nor log segments found. Creating new log.
Sep 25 22:54:39 dist-test-slave-x1lh krb5kdc[24993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:39.127832 25073 log.cc:824] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:39.129752 25073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874: No bootstrap required, opened a new log
I20260925 22:54:39.132423 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:39.119927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38060 (local address 127.20.114.254:40107)
0925 22:54:39.120336 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:39.120349 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:39.120383 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:54:39.121162 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:39.121180 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:39.121495 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:39.121824 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:39.122621 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.123567 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.124419 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.125013 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.125180 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:39.127819 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:39.127827 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:39.127846 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:39.127873 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:39.129734 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:39.130185 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:39.130197 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:39.130202 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:39.130310 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:39.130603 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:39.130606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:39.130608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:39.130983 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:39.131179 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:39.131462 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 22:54:39.131743 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":129,"threads_started":1}
I20260925 22:54:39.133472 25073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07eb2420aa4b47d1989ce310777b2874" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40107 } }
I20260925 22:54:39.133742 25073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:39.133805 25073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07eb2420aa4b47d1989ce310777b2874, State: Initialized, Role: FOLLOWER
I20260925 22:54:39.134254 25073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07eb2420aa4b47d1989ce310777b2874" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40107 } }
I20260925 22:54:39.134407 25073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:54:39.134506 25073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:54:39.134637 25073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:39.135592 25073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07eb2420aa4b47d1989ce310777b2874" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40107 } }
I20260925 22:54:39.135944 25073 leader_election.cc:302] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07eb2420aa4b47d1989ce310777b2874; no voters:
I20260925 22:54:39.136276 25073 leader_election.cc:288] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:54:39.136348 25078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:39.136627 25078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874 [term 1 LEADER]: Becoming Leader. State: Replica: 07eb2420aa4b47d1989ce310777b2874, State: Running, Role: LEADER
I20260925 22:54:39.137001 25078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07eb2420aa4b47d1989ce310777b2874" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40107 } }
I20260925 22:54:39.137588 25073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:39.138675 25080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07eb2420aa4b47d1989ce310777b2874. Latest consensus state: current_term: 1 leader_uuid: "07eb2420aa4b47d1989ce310777b2874" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07eb2420aa4b47d1989ce310777b2874" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40107 } } }
I20260925 22:54:39.138795 25080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:39.138937 25079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07eb2420aa4b47d1989ce310777b2874" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07eb2420aa4b47d1989ce310777b2874" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40107 } } }
I20260925 22:54:39.139098 25079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:39.139362 25088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:54:39.142151 25088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:54:39.147125 25088 catalog_manager.cc:1347] Generated new cluster ID: dd0874277aa744ff9aad5d8357c4e4ed
I20260925 22:54:39.147193 25088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:39.175060 25088 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:54:39.176018 25088 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:39.187071 25088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874: Generated new TSK 0
I20260925 22:54:39.187909 25088 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.114.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
I20260925 22:54:39.228225 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:40107
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:39.348240 25099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:39.348495 25099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:39.348574 25099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:39.352617 25099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:39.352679 25099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:39.352782 25099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:54:39.358152 25099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:40107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:39.359212 25099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:39.360239 25099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:39.371347 25107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:39.371348 25105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:39.371336 25104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:39.371572 25099 server_base.cc:1034] running on GCE node
I20260925 22:54:39.372035 25099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:39.372704 25099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:39.373886 25099 hybrid_clock.cc:648] HybridClock initialized: now 1790376879373854 us; error 54 us; skew 500 ppm
Sep 25 22:54:39 dist-test-slave-x1lh krb5kdc[24993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376879, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:39.376667 25099 init.cc:377] Logged in from keytab as kudu/127.20.114.193@KRBTEST.COM (short username kudu)
I20260925 22:54:39.377400 25099 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:39.377462 25099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:39.377627 25099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:39.379753 25099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "c2671a1afcdb489c973f040657c506df"
format_stamp: "Formatted at 2026-09-25 22:54:39 on dist-test-slave-x1lh"
server_key: "ada0dde794ca1d42f29aa49153066977"
server_key_iv: "d6619a43a3852a06e88f47fa05d048e9"
server_key_version: "encryptionkey@0"
I20260925 22:54:39.380264 25099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "c2671a1afcdb489c973f040657c506df"
format_stamp: "Formatted at 2026-09-25 22:54:39 on dist-test-slave-x1lh"
server_key: "ada0dde794ca1d42f29aa49153066977"
server_key_iv: "d6619a43a3852a06e88f47fa05d048e9"
server_key_version: "encryptionkey@0"
I20260925 22:54:39.383827 25099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:54:39.386164 25113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:39.387357 25099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:54:39.387447 25099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "c2671a1afcdb489c973f040657c506df"
format_stamp: "Formatted at 2026-09-25 22:54:39 on dist-test-slave-x1lh"
server_key: "ada0dde794ca1d42f29aa49153066977"
server_key_iv: "d6619a43a3852a06e88f47fa05d048e9"
server_key_version: "encryptionkey@0"
I20260925 22:54:39.387526 25099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:39.402070 25099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:39.405349 25099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:39.405494 25099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:39.406000 25099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:39.406790 25099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:39.406834 25099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:39.406872 25099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:39.406888 25099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:39.415577 25099 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:35779
I20260925 22:54:39.415604 25226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:35779 every 8 connection(s)
I20260925 22:54:39.416184 25099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
Sep 25 22:54:39 dist-test-slave-x1lh krb5kdc[24993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376879, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:39.425216 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 25099
I20260925 22:54:39.425352 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:54:39.425634 20939 external_mini_cluster.cc:1428] Setting key 878af7cdbee03768d8b08ebb792c435d
I20260925 22:54:39.429210 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:39.417976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:38305 (local address 127.20.114.254:40107)
0925 22:54:39.418147 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:39.418152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:39.418967 (+ 815us) server_negotiation.cc:407] Connection header received
0925 22:54:39.420167 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:39.420170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:39.420215 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:39.420289 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:39.421465 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.421992 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.423037 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.423279 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.423354 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:39.425772 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:39.425775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:39.425777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:39.425799 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:39.427218 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:39.427727 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:39.427730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:39.427732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:39.427812 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:39.428162 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:39.428165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:39.428167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:39.428335 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:39.428441 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:39.428928 (+ 487us) server_negotiation.cc:299] Negotiation successful
0925 22:54:39.429049 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 22:54:39.429986 25229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:39.418377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:40107 (local address 127.20.114.193:38305)
0925 22:54:39.418818 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:39.418861 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:39.419985 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:39.420448 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:39.420457 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:39.420796 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:39.421345 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:39.421358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.422091 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.422094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:39.422933 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:39.422941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.423399 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.423402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:39.423544 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:39.424053 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:39.424081 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:39.425610 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:39.427349 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:39.427355 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:39.427369 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:39.427622 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:39.427911 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:39.427914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:39.427916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:39.428059 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:39.428440 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:39.428446 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:39.428679 (+ 233us) client_negotiation.cc:769] Sending connection context
0925 22:54:39.428863 (+ 184us) client_negotiation.cc:240] Negotiation successful
0925 22:54:39.429192 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":113,"threads_started":1}
I20260925 22:54:39.431214 25227 heartbeater.cc:347] Connected to a master server at 127.20.114.254:40107
I20260925 22:54:39.431406 25227 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:39.431787 25227 heartbeater.cc:511] Master 127.20.114.254:40107 requested a full tablet report, sending...
I20260925 22:54:39.433228 25037 ts_manager.cc:194] Registered new tserver with Master: c2671a1afcdb489c973f040657c506df (127.20.114.193:35779)
I20260925 22:54:39.434271 25037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:38305
WARNING: no policy specified for kudu/127.20.114.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
I20260925 22:54:39.452824 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:40107
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:39.563560 25232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:39.563810 25232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:39.563855 25232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:39.568009 25232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:39.568068 25232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:39.568166 25232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:54:39.573442 25232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:40107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:39.574617 25232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:39.575623 25232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:39.584456 25237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:39.584514 25240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:39.584476 25238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:39.584802 25232 server_base.cc:1034] running on GCE node
I20260925 22:54:39.585225 25232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:39.585913 25232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:39.587114 25232 hybrid_clock.cc:648] HybridClock initialized: now 1790376879587098 us; error 27 us; skew 500 ppm
Sep 25 22:54:39 dist-test-slave-x1lh krb5kdc[24993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376879, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:39.589989 25232 init.cc:377] Logged in from keytab as kudu/127.20.114.194@KRBTEST.COM (short username kudu)
I20260925 22:54:39.590803 25232 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:39.590896 25232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:39.591144 25232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:39.593319 25232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "124b9d04541f49459effad6f4cbe18bf"
format_stamp: "Formatted at 2026-09-25 22:54:39 on dist-test-slave-x1lh"
server_key: "450352e749c89ceb7c8aef1f56024a4a"
server_key_iv: "417a3c55b3653c09a9cc9657c1c281bb"
server_key_version: "encryptionkey@0"
I20260925 22:54:39.593906 25232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "124b9d04541f49459effad6f4cbe18bf"
format_stamp: "Formatted at 2026-09-25 22:54:39 on dist-test-slave-x1lh"
server_key: "450352e749c89ceb7c8aef1f56024a4a"
server_key_iv: "417a3c55b3653c09a9cc9657c1c281bb"
server_key_version: "encryptionkey@0"
I20260925 22:54:39.597450 25232 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:54:39.599807 25246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:39.601133 25232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:39.601257 25232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "124b9d04541f49459effad6f4cbe18bf"
format_stamp: "Formatted at 2026-09-25 22:54:39 on dist-test-slave-x1lh"
server_key: "450352e749c89ceb7c8aef1f56024a4a"
server_key_iv: "417a3c55b3653c09a9cc9657c1c281bb"
server_key_version: "encryptionkey@0"
I20260925 22:54:39.601378 25232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:39.631587 25232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:39.634644 25232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:39.634845 25232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:39.635412 25232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:39.636264 25232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:39.636341 25232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:39.636418 25232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:39.636471 25232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:39.645627 25232 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:42175
I20260925 22:54:39.645646 25359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:42175 every 8 connection(s)
I20260925 22:54:39.646238 25232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:54:39.649080 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 25232
I20260925 22:54:39.649204 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:54:39.649572 20939 external_mini_cluster.cc:1428] Setting key 6f2978cd63e2b6c156a0c5357c286060
Sep 25 22:54:39 dist-test-slave-x1lh krb5kdc[24993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376879, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.114.195@KRBTEST.COM; defaulting to no policy
I20260925 22:54:39.660733 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:39.647871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:51715 (local address 127.20.114.254:40107)
0925 22:54:39.648034 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:39.648040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:39.648909 (+ 869us) server_negotiation.cc:407] Connection header received
0925 22:54:39.650402 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:39.650406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:39.650453 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:39.650539 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:39.651916 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.652434 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.653747 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.654163 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.654289 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:39.657043 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:39.657049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:39.657052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:39.657080 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:39.658716 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:39.659236 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:39.659240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:39.659242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:39.659325 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:39.659680 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:39.659687 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:39.659690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:39.659909 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:39.659995 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:39.660503 (+ 508us) server_negotiation.cc:299] Negotiation successful
0925 22:54:39.660616 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:54:39.661624 25362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:39.648287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:40107 (local address 127.20.114.194:51715)
0925 22:54:39.648732 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:39.648774 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:39.650215 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:39.650695 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:39.650702 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:39.651091 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:39.651763 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:39.651785 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.652620 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.652626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:39.653626 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:39.653635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.654305 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.654308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:39.654442 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:39.655101 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:39.655125 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:39.656789 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:39.658858 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:39.658862 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:39.658873 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:39.659121 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:39.659410 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:39.659412 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:39.659413 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:39.659589 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:39.660003 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:39.660007 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:39.660232 (+ 225us) client_negotiation.cc:769] Sending connection context
0925 22:54:39.660375 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 22:54:39.660721 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.195@KRBTEST.COM" created.
I20260925 22:54:39.662842 25360 heartbeater.cc:347] Connected to a master server at 127.20.114.254:40107
I20260925 22:54:39.663102 25360 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:39.663568 25360 heartbeater.cc:511] Master 127.20.114.254:40107 requested a full tablet report, sending...
I20260925 22:54:39.664752 25037 ts_manager.cc:194] Registered new tserver with Master: 124b9d04541f49459effad6f4cbe18bf (127.20.114.194:42175)
I20260925 22:54:39.665642 25037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:51715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
I20260925 22:54:39.677079 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:40107
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:39.785410 25365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:39.785661 25365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:39.785704 25365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:39.789637 25365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:39.789683 25365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:39.789740 25365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:54:39.794945 25365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:40107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:39.795857 25365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:39.796764 25365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:39.806121 25371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:39.806227 25365 server_base.cc:1034] running on GCE node
W20260925 22:54:39.806144 25370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:39.806384 25373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:39.806753 25365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:39.807394 25365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:39.808571 25365 hybrid_clock.cc:648] HybridClock initialized: now 1790376879808548 us; error 35 us; skew 500 ppm
Sep 25 22:54:39 dist-test-slave-x1lh krb5kdc[24993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376879, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:39.811391 25365 init.cc:377] Logged in from keytab as kudu/127.20.114.195@KRBTEST.COM (short username kudu)
I20260925 22:54:39.812152 25365 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:39.812250 25365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:39.812475 25365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:39.814522 25365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "9c6a00a3d1474c6ba6fcaed9710f1770"
format_stamp: "Formatted at 2026-09-25 22:54:39 on dist-test-slave-x1lh"
server_key: "b0400159b42140a62264afdd2fb77d7a"
server_key_iv: "b75b1073b83eba68d4673e76f09b459f"
server_key_version: "encryptionkey@0"
I20260925 22:54:39.815071 25365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "9c6a00a3d1474c6ba6fcaed9710f1770"
format_stamp: "Formatted at 2026-09-25 22:54:39 on dist-test-slave-x1lh"
server_key: "b0400159b42140a62264afdd2fb77d7a"
server_key_iv: "b75b1073b83eba68d4673e76f09b459f"
server_key_version: "encryptionkey@0"
I20260925 22:54:39.818646 25365 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 22:54:39.820945 25379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:39.822302 25365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:54:39.822429 25365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "9c6a00a3d1474c6ba6fcaed9710f1770"
format_stamp: "Formatted at 2026-09-25 22:54:39 on dist-test-slave-x1lh"
server_key: "b0400159b42140a62264afdd2fb77d7a"
server_key_iv: "b75b1073b83eba68d4673e76f09b459f"
server_key_version: "encryptionkey@0"
I20260925 22:54:39.822554 25365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:39.842597 25365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:39.846043 25365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:39.846251 25365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:39.846838 25365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:39.847661 25365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:39.847731 25365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:39.847805 25365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:39.847846 25365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:39.856302 25365 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:45577
I20260925 22:54:39.856321 25492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:45577 every 8 connection(s)
I20260925 22:54:39.856914 25365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:54:39.862799 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 25365
I20260925 22:54:39.862936 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:54:39.863227 20939 external_mini_cluster.cc:1428] Setting key 9a6a2b739e0b6a8c084e85f7059d5750
Sep 25 22:54:39 dist-test-slave-x1lh krb5kdc[24993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376879, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:39.869988 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:39.858608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:39371 (local address 127.20.114.254:40107)
0925 22:54:39.858750 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:39.858754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:39.859588 (+ 834us) server_negotiation.cc:407] Connection header received
0925 22:54:39.860791 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:39.860794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:39.860839 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:39.860962 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:39.862057 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.862615 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.863926 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.864155 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.864268 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:39.866743 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:39.866747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:39.866749 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:39.866780 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:39.867974 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:39.868469 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:39.868473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:39.868475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:39.868558 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:39.868897 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:39.868900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:39.868902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:39.869093 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:39.869200 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:39.869708 (+ 508us) server_negotiation.cc:299] Negotiation successful
0925 22:54:39.869826 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:54:39.870749 25495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:39.858992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:40107 (local address 127.20.114.195:39371)
0925 22:54:39.859428 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:39.859471 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:39.860602 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:39.861125 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:39.861131 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:39.861467 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:39.861938 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:39.861950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.862734 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.862737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:39.863799 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:39.863811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.864300 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.864305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:39.864480 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:39.865120 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:39.865136 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:39.866587 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:39.868105 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:39.868110 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:39.868124 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:39.868365 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:39.868648 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:39.868653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:39.868655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:39.868787 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:39.869205 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:39.869213 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:39.869460 (+ 247us) client_negotiation.cc:769] Sending connection context
0925 22:54:39.869664 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 22:54:39.869966 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":112,"threads_started":1}
I20260925 22:54:39.871757 25493 heartbeater.cc:347] Connected to a master server at 127.20.114.254:40107
I20260925 22:54:39.871990 25493 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:39.872416 25493 heartbeater.cc:511] Master 127.20.114.254:40107 requested a full tablet report, sending...
I20260925 22:54:39.873406 25037 ts_manager.cc:194] Registered new tserver with Master: 9c6a00a3d1474c6ba6fcaed9710f1770 (127.20.114.195:45577)
I20260925 22:54:39.873908 25037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:39371
I20260925 22:54:39.876943 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:54:39.885366 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:39.878717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38076 (local address 127.20.114.254:40107)
0925 22:54:39.878829 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:39.878832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:39.879032 (+ 200us) server_negotiation.cc:407] Connection header received
0925 22:54:39.879118 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:39.879120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:39.879153 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:39.879216 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:39.879847 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.880323 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.881135 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.881336 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.881410 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:39.882611 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:39.882614 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:39.882616 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:39.882636 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:39.883822 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:39.884191 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:39.884193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:39.884195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:39.884261 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:39.884552 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:39.884554 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:39.884555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:39.884756 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:39.884857 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:39.885166 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 22:54:39.885256 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:54:39.887915 25037 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:54:39.891072 25037 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:54:39.905047 25504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:39.899332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:45577 (local address 127.0.0.1:58112)
0925 22:54:39.899644 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:39.899696 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:39.899831 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:39.900456 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:39.900464 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:39.900488 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:39.900672 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:39.900683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.902390 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.902396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:39.903447 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:39.903458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.904457 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.904461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:39.904588 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:39.904629 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 22:54:39.904754 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 22:54:39.904871 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":1,"thread_start_us":62,"threads_started":1}
I20260925 22:54:39.905055 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:39.899453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58112 (local address 127.20.114.195:45577)
0925 22:54:39.899821 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:39.899832 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:39.899857 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:54:39.900001 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:39.900008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:39.900283 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:39.901213 (+ 930us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:39.901358 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.902264 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.903579 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.904330 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.904473 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:39.904605 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:39.904783 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 22:54:39.904898 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":167,"threads_started":1}
I20260925 22:54:39.906092 25506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:39.900028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:42175 (local address 127.0.0.1:53534)
0925 22:54:39.900283 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:39.900297 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:39.900391 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:39.901522 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:39.901525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:39.901537 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:39.901694 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:39.901705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.903125 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.903129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:39.904538 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:39.904550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.905709 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.905712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:39.905797 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:39.905819 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:54:39.905900 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 22:54:39.905960 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":179,"thread_start_us":89,"threads_started":1}
I20260925 22:54:39.906178 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:39.900196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.20.114.194:42175)
0925 22:54:39.901085 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:39.901096 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:39.901128 (+ 32us) server_negotiation.cc:407] Connection header received
0925 22:54:39.901204 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:39.901209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:39.901371 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:39.901685 (+ 314us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:39.901851 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.902981 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.904686 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.905579 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.905712 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:39.905831 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:39.905926 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 22:54:39.906020 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":123,"threads_started":1}
I20260925 22:54:39.906481 25508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:39.900421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:35779 (local address 127.0.0.1:59588)
0925 22:54:39.900642 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:39.900655 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:39.900735 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:39.901481 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:39.901486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:39.901506 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:39.901664 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:39.901672 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.903167 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.903170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:39.904546 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:39.904552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.906112 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.906116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:39.906206 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:39.906228 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:54:39.906283 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 22:54:39.906345 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":101,"spinlock_wait_cycles":768,"thread_start_us":83,"threads_started":1}
I20260925 22:54:39.907589 25509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:39.900715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59588 (local address 127.20.114.193:35779)
0925 22:54:39.901014 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:39.901022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:39.901059 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:54:39.901137 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:39.901146 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:39.901333 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:39.901487 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:39.901799 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.902981 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.904686 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.905970 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.907081 (+ 1111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:39.907309 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:39.907357 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 22:54:39.907444 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":109,"threads_started":1}
I20260925 22:54:39.908293 25427 tablet_service.cc:1467] Processing CreateTablet for tablet 57f20d8457e948a0947dc7d23ea7323e (DEFAULT_TABLE table=test-table [id=343edacbea7a438993b8efd9fd65d0b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:39.908851 25294 tablet_service.cc:1467] Processing CreateTablet for tablet 57f20d8457e948a0947dc7d23ea7323e (DEFAULT_TABLE table=test-table [id=343edacbea7a438993b8efd9fd65d0b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:39.909461 25427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f20d8457e948a0947dc7d23ea7323e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:39.909797 25294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f20d8457e948a0947dc7d23ea7323e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:39.910069 25161 tablet_service.cc:1467] Processing CreateTablet for tablet 57f20d8457e948a0947dc7d23ea7323e (DEFAULT_TABLE table=test-table [id=343edacbea7a438993b8efd9fd65d0b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:39.911022 25161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f20d8457e948a0947dc7d23ea7323e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:39.915963 25510 tablet_bootstrap.cc:492] T 57f20d8457e948a0947dc7d23ea7323e P 124b9d04541f49459effad6f4cbe18bf: Bootstrap starting.
I20260925 22:54:39.916896 25511 tablet_bootstrap.cc:492] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770: Bootstrap starting.
I20260925 22:54:39.918133 25510 tablet_bootstrap.cc:654] T 57f20d8457e948a0947dc7d23ea7323e P 124b9d04541f49459effad6f4cbe18bf: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:39.918776 25511 tablet_bootstrap.cc:654] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:39.919173 25510 log.cc:824] T 57f20d8457e948a0947dc7d23ea7323e P 124b9d04541f49459effad6f4cbe18bf: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:39.919749 25512 tablet_bootstrap.cc:492] T 57f20d8457e948a0947dc7d23ea7323e P c2671a1afcdb489c973f040657c506df: Bootstrap starting.
I20260925 22:54:39.919880 25511 log.cc:824] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:39.921159 25510 tablet_bootstrap.cc:492] T 57f20d8457e948a0947dc7d23ea7323e P 124b9d04541f49459effad6f4cbe18bf: No bootstrap required, opened a new log
I20260925 22:54:39.921353 25510 ts_tablet_manager.cc:1397] T 57f20d8457e948a0947dc7d23ea7323e P 124b9d04541f49459effad6f4cbe18bf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:54:39.921801 25512 tablet_bootstrap.cc:654] T 57f20d8457e948a0947dc7d23ea7323e P c2671a1afcdb489c973f040657c506df: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:39.921887 25511 tablet_bootstrap.cc:492] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770: No bootstrap required, opened a new log
I20260925 22:54:39.922113 25511 ts_tablet_manager.cc:1397] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:54:39.922953 25512 log.cc:824] T 57f20d8457e948a0947dc7d23ea7323e P c2671a1afcdb489c973f040657c506df: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:39.924443 25510 raft_consensus.cc:348] T 57f20d8457e948a0947dc7d23ea7323e P 124b9d04541f49459effad6f4cbe18bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a00a3d1474c6ba6fcaed9710f1770" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45577 } } peers { permanent_uuid: "124b9d04541f49459effad6f4cbe18bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42175 } } peers { permanent_uuid: "c2671a1afcdb489c973f040657c506df" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35779 } }
I20260925 22:54:39.924683 25510 raft_consensus.cc:374] T 57f20d8457e948a0947dc7d23ea7323e P 124b9d04541f49459effad6f4cbe18bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:39.924755 25510 raft_consensus.cc:729] T 57f20d8457e948a0947dc7d23ea7323e P 124b9d04541f49459effad6f4cbe18bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124b9d04541f49459effad6f4cbe18bf, State: Initialized, Role: FOLLOWER
I20260925 22:54:39.924782 25512 tablet_bootstrap.cc:492] T 57f20d8457e948a0947dc7d23ea7323e P c2671a1afcdb489c973f040657c506df: No bootstrap required, opened a new log
I20260925 22:54:39.924973 25512 ts_tablet_manager.cc:1397] T 57f20d8457e948a0947dc7d23ea7323e P c2671a1afcdb489c973f040657c506df: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:54:39.925235 25510 consensus_queue.cc:260] T 57f20d8457e948a0947dc7d23ea7323e P 124b9d04541f49459effad6f4cbe18bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a00a3d1474c6ba6fcaed9710f1770" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45577 } } peers { permanent_uuid: "124b9d04541f49459effad6f4cbe18bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42175 } } peers { permanent_uuid: "c2671a1afcdb489c973f040657c506df" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35779 } }
I20260925 22:54:39.925817 25360 heartbeater.cc:503] Master 127.20.114.254:40107 was elected leader, sending a full tablet report...
I20260925 22:54:39.925769 25511 raft_consensus.cc:348] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a00a3d1474c6ba6fcaed9710f1770" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45577 } } peers { permanent_uuid: "124b9d04541f49459effad6f4cbe18bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42175 } } peers { permanent_uuid: "c2671a1afcdb489c973f040657c506df" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35779 } }
I20260925 22:54:39.926065 25511 raft_consensus.cc:374] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:39.926129 25511 raft_consensus.cc:729] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6a00a3d1474c6ba6fcaed9710f1770, State: Initialized, Role: FOLLOWER
I20260925 22:54:39.926245 25510 ts_tablet_manager.cc:1428] T 57f20d8457e948a0947dc7d23ea7323e P 124b9d04541f49459effad6f4cbe18bf: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:54:39.926571 25511 consensus_queue.cc:260] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a00a3d1474c6ba6fcaed9710f1770" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45577 } } peers { permanent_uuid: "124b9d04541f49459effad6f4cbe18bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42175 } } peers { permanent_uuid: "c2671a1afcdb489c973f040657c506df" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35779 } }
I20260925 22:54:39.927239 25493 heartbeater.cc:503] Master 127.20.114.254:40107 was elected leader, sending a full tablet report...
I20260925 22:54:39.928380 25511 ts_tablet_manager.cc:1428] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:54:39.929663 25512 raft_consensus.cc:348] T 57f20d8457e948a0947dc7d23ea7323e P c2671a1afcdb489c973f040657c506df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a00a3d1474c6ba6fcaed9710f1770" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45577 } } peers { permanent_uuid: "124b9d04541f49459effad6f4cbe18bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42175 } } peers { permanent_uuid: "c2671a1afcdb489c973f040657c506df" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35779 } }
I20260925 22:54:39.929900 25512 raft_consensus.cc:374] T 57f20d8457e948a0947dc7d23ea7323e P c2671a1afcdb489c973f040657c506df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:39.929961 25512 raft_consensus.cc:729] T 57f20d8457e948a0947dc7d23ea7323e P c2671a1afcdb489c973f040657c506df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2671a1afcdb489c973f040657c506df, State: Initialized, Role: FOLLOWER
I20260925 22:54:39.930361 25512 consensus_queue.cc:260] T 57f20d8457e948a0947dc7d23ea7323e P c2671a1afcdb489c973f040657c506df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a00a3d1474c6ba6fcaed9710f1770" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45577 } } peers { permanent_uuid: "124b9d04541f49459effad6f4cbe18bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42175 } } peers { permanent_uuid: "c2671a1afcdb489c973f040657c506df" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35779 } }
I20260925 22:54:39.931129 25227 heartbeater.cc:503] Master 127.20.114.254:40107 was elected leader, sending a full tablet report...
I20260925 22:54:39.931227 25512 ts_tablet_manager.cc:1428] T 57f20d8457e948a0947dc7d23ea7323e P c2671a1afcdb489c973f040657c506df: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:54:39.972990 25517 raft_consensus.cc:482] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:54:39.973209 25517 raft_consensus.cc:504] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a00a3d1474c6ba6fcaed9710f1770" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45577 } } peers { permanent_uuid: "124b9d04541f49459effad6f4cbe18bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42175 } } peers { permanent_uuid: "c2671a1afcdb489c973f040657c506df" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35779 } }
I20260925 22:54:39.974360 25517 leader_election.cc:288] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 124b9d04541f49459effad6f4cbe18bf (127.20.114.194:42175), c2671a1afcdb489c973f040657c506df (127.20.114.193:35779)
I20260925 22:54:39.978674 25495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:39.974653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:42175 (local address 127.20.114.195:55077)
0925 22:54:39.974781 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:39.974798 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:39.974887 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:39.975292 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:39.975298 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:39.975323 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:39.975455 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:39.975465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.976377 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.976380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:39.977478 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:39.977487 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.978236 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.978239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:39.978340 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:39.978366 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:54:39.978451 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 22:54:39.978529 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 22:54:39.978677 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:39.974794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:55077 (local address 127.20.114.194:42175)
0925 22:54:39.975077 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:39.975082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:39.975096 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:54:39.975147 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:39.975153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:39.975205 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:39.975265 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:39.975580 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.976276 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.977587 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.978130 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.978277 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:39.978374 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:39.978469 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 22:54:39.978548 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260925 22:54:39.979044 25509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:39.975033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:38561 (local address 127.20.114.193:35779)
0925 22:54:39.975177 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:39.975181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:39.975508 (+ 327us) server_negotiation.cc:407] Connection header received
0925 22:54:39.975640 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:39.975643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:39.975692 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:39.975766 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:39.976027 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.976661 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.977945 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.978471 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.978575 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:39.978621 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:39.978838 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 22:54:39.978914 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 22:54:39.979378 25519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:39.974903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:35779 (local address 127.20.114.195:38561)
0925 22:54:39.975411 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:39.975427 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:39.975527 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:39.975799 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:39.975802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:39.975822 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:39.975932 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:39.975938 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.976770 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.976773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:39.977829 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:39.977840 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:39.978575 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:39.978579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:39.978678 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:39.978711 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 22:54:39.979198 (+ 487us) client_negotiation.cc:240] Negotiation successful
0925 22:54:39.979266 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":128,"threads_started":1}
I20260925 22:54:39.979396 25314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f20d8457e948a0947dc7d23ea7323e" candidate_uuid: "9c6a00a3d1474c6ba6fcaed9710f1770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124b9d04541f49459effad6f4cbe18bf" is_pre_election: true
I20260925 22:54:39.979724 25314 raft_consensus.cc:2454] T 57f20d8457e948a0947dc7d23ea7323e P 124b9d04541f49459effad6f4cbe18bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c6a00a3d1474c6ba6fcaed9710f1770 in term 0.
I20260925 22:54:39.980057 25181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f20d8457e948a0947dc7d23ea7323e" candidate_uuid: "9c6a00a3d1474c6ba6fcaed9710f1770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2671a1afcdb489c973f040657c506df" is_pre_election: true
I20260925 22:54:39.980192 25383 leader_election.cc:302] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124b9d04541f49459effad6f4cbe18bf, 9c6a00a3d1474c6ba6fcaed9710f1770; no voters:
I20260925 22:54:39.980347 25181 raft_consensus.cc:2454] T 57f20d8457e948a0947dc7d23ea7323e P c2671a1afcdb489c973f040657c506df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c6a00a3d1474c6ba6fcaed9710f1770 in term 0.
I20260925 22:54:39.980475 25517 raft_consensus.cc:2781] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:54:39.980552 25517 raft_consensus.cc:482] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:54:39.980584 25517 raft_consensus.cc:3037] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:39.981660 25517 raft_consensus.cc:504] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a00a3d1474c6ba6fcaed9710f1770" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45577 } } peers { permanent_uuid: "124b9d04541f49459effad6f4cbe18bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42175 } } peers { permanent_uuid: "c2671a1afcdb489c973f040657c506df" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35779 } }
I20260925 22:54:39.981974 25517 leader_election.cc:288] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770 [CANDIDATE]: Term 1 election: Requested vote from peers 124b9d04541f49459effad6f4cbe18bf (127.20.114.194:42175), c2671a1afcdb489c973f040657c506df (127.20.114.193:35779)
I20260925 22:54:39.982312 25314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f20d8457e948a0947dc7d23ea7323e" candidate_uuid: "9c6a00a3d1474c6ba6fcaed9710f1770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124b9d04541f49459effad6f4cbe18bf"
I20260925 22:54:39.982417 25181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f20d8457e948a0947dc7d23ea7323e" candidate_uuid: "9c6a00a3d1474c6ba6fcaed9710f1770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2671a1afcdb489c973f040657c506df"
I20260925 22:54:39.982440 25314 raft_consensus.cc:3037] T 57f20d8457e948a0947dc7d23ea7323e P 124b9d04541f49459effad6f4cbe18bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:39.982571 25181 raft_consensus.cc:3037] T 57f20d8457e948a0947dc7d23ea7323e P c2671a1afcdb489c973f040657c506df [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:39.983496 25314 raft_consensus.cc:2454] T 57f20d8457e948a0947dc7d23ea7323e P 124b9d04541f49459effad6f4cbe18bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c6a00a3d1474c6ba6fcaed9710f1770 in term 1.
I20260925 22:54:39.983568 25181 raft_consensus.cc:2454] T 57f20d8457e948a0947dc7d23ea7323e P c2671a1afcdb489c973f040657c506df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c6a00a3d1474c6ba6fcaed9710f1770 in term 1.
I20260925 22:54:39.983789 25383 leader_election.cc:302] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124b9d04541f49459effad6f4cbe18bf, 9c6a00a3d1474c6ba6fcaed9710f1770; no voters:
I20260925 22:54:39.983948 25517 raft_consensus.cc:2781] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:39.984146 25517 raft_consensus.cc:686] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770 [term 1 LEADER]: Becoming Leader. State: Replica: 9c6a00a3d1474c6ba6fcaed9710f1770, State: Running, Role: LEADER
I20260925 22:54:39.984462 25517 consensus_queue.cc:237] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a00a3d1474c6ba6fcaed9710f1770" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45577 } } peers { permanent_uuid: "124b9d04541f49459effad6f4cbe18bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42175 } } peers { permanent_uuid: "c2671a1afcdb489c973f040657c506df" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35779 } }
I20260925 22:54:39.987649 25037 catalog_manager.cc:5495] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c6a00a3d1474c6ba6fcaed9710f1770 (127.20.114.195). New cstate: current_term: 1 leader_uuid: "9c6a00a3d1474c6ba6fcaed9710f1770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a00a3d1474c6ba6fcaed9710f1770" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "124b9d04541f49459effad6f4cbe18bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2671a1afcdb489c973f040657c506df" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35779 } health_report { overall_health: UNKNOWN } } }
I20260925 22:54:40.002341 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:39.999004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58126 (local address 127.20.114.195:45577)
0925 22:54:39.999134 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:39.999139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:39.999158 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:54:39.999294 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:39.999296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:39.999348 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:39.999431 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:39.999658 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:40.000221 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:40.001092 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:40.001290 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:40.001365 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:40.001598 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:40.001990 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:40.002085 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:40.002119 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 22:54:40.002161 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:54:40.008250 25314 raft_consensus.cc:1261] T 57f20d8457e948a0947dc7d23ea7323e P 124b9d04541f49459effad6f4cbe18bf [term 1 FOLLOWER]: Refusing update from remote peer 9c6a00a3d1474c6ba6fcaed9710f1770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:40.008250 25181 raft_consensus.cc:1261] T 57f20d8457e948a0947dc7d23ea7323e P c2671a1afcdb489c973f040657c506df [term 1 FOLLOWER]: Refusing update from remote peer 9c6a00a3d1474c6ba6fcaed9710f1770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:40.008940 25517 consensus_queue.cc:1035] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124b9d04541f49459effad6f4cbe18bf" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:40.009131 25520 consensus_queue.cc:1035] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2671a1afcdb489c973f040657c506df" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:40.017894 25525 mvcc.cc:205] Tried to move back new op lower bound from 7333383700506075136 to 7333383700419268608. Current Snapshot: MvccSnapshot[applied={T|T < 7333383700506075136}]
I20260925 22:54:40.018726 25524 mvcc.cc:205] Tried to move back new op lower bound from 7333383700506075136 to 7333383700419268608. Current Snapshot: MvccSnapshot[applied={T|T < 7333383700506075136}]
I20260925 22:54:40.024398 25037 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:40.024577 25037 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:40.027004 25037 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07eb2420aa4b47d1989ce310777b2874: Sending DeleteTablet for 3 replicas of tablet 57f20d8457e948a0947dc7d23ea7323e
I20260925 22:54:40.027792 25427 tablet_service.cc:1514] Processing DeleteTablet for tablet 57f20d8457e948a0947dc7d23ea7323e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:40 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:58112
I20260925 22:54:40.027868 25161 tablet_service.cc:1514] Processing DeleteTablet for tablet 57f20d8457e948a0947dc7d23ea7323e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:40 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:59588
I20260925 22:54:40.028262 25533 tablet_replica.cc:330] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770: stopping tablet replica
I20260925 22:54:40.028263 25294 tablet_service.cc:1514] Processing DeleteTablet for tablet 57f20d8457e948a0947dc7d23ea7323e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:40 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:53534
I20260925 22:54:40.028573 25533 raft_consensus.cc:2229] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:54:40.028735 25535 tablet_replica.cc:330] T 57f20d8457e948a0947dc7d23ea7323e P 124b9d04541f49459effad6f4cbe18bf: stopping tablet replica
I20260925 22:54:40.028976 25533 raft_consensus.cc:2258] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:40.029150 25535 raft_consensus.cc:2229] T 57f20d8457e948a0947dc7d23ea7323e P 124b9d04541f49459effad6f4cbe18bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:40.029277 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 25099
I20260925 22:54:40.030009 25535 raft_consensus.cc:2258] T 57f20d8457e948a0947dc7d23ea7323e P 124b9d04541f49459effad6f4cbe18bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:40.031648 25535 ts_tablet_manager.cc:1905] T 57f20d8457e948a0947dc7d23ea7323e P 124b9d04541f49459effad6f4cbe18bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:40.032234 25533 ts_tablet_manager.cc:1905] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:40.034865 25533 ts_tablet_manager.cc:1918] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:54:40.034960 25533 log.cc:1196] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/57f20d8457e948a0947dc7d23ea7323e
I20260925 22:54:40.035210 25535 ts_tablet_manager.cc:1918] T 57f20d8457e948a0947dc7d23ea7323e P 124b9d04541f49459effad6f4cbe18bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:54:40.035279 25533 ts_tablet_manager.cc:1939] T 57f20d8457e948a0947dc7d23ea7323e P 9c6a00a3d1474c6ba6fcaed9710f1770: Deleting consensus metadata
I20260925 22:54:40.035390 25535 log.cc:1196] T 57f20d8457e948a0947dc7d23ea7323e P 124b9d04541f49459effad6f4cbe18bf: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/57f20d8457e948a0947dc7d23ea7323e
I20260925 22:54:40.035753 25535 ts_tablet_manager.cc:1939] T 57f20d8457e948a0947dc7d23ea7323e P 124b9d04541f49459effad6f4cbe18bf: Deleting consensus metadata
I20260925 22:54:40.036402 25024 catalog_manager.cc:4841] TS 9c6a00a3d1474c6ba6fcaed9710f1770 (127.20.114.195:45577): tablet 57f20d8457e948a0947dc7d23ea7323e (table test-table [id=343edacbea7a438993b8efd9fd65d0b0]) successfully deleted
I20260925 22:54:40.037169 25024 catalog_manager.cc:4841] TS 124b9d04541f49459effad6f4cbe18bf (127.20.114.194:42175): tablet 57f20d8457e948a0947dc7d23ea7323e (table test-table [id=343edacbea7a438993b8efd9fd65d0b0]) successfully deleted
W20260925 22:54:40.038100 25024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.114.193:35779 (error 108)
I20260925 22:54:40.038317 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 25232
I20260925 22:54:40.039746 25508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:40.039357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:35779 (local address 127.0.0.1:59592)
0925 22:54:40.039557 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:40.039641 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.193:35779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260925 22:54:40.039917 25024 catalog_manager.cc:4584] TS c2671a1afcdb489c973f040657c506df (127.20.114.193:35779): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 57f20d8457e948a0947dc7d23ea7323e: Network error: Client connection negotiation failed: client connection to 127.20.114.193:35779: connect: Connection refused (error 111)
I20260925 22:54:40.045207 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 25365
I20260925 22:54:40.051102 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 25006
2026-09-25T22:54:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2257 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T22:54:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:54:40Z Disabled control of system clock
I20260925 22:54:40.081231 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:44897
--webserver_interface=127.20.114.254
--webserver_port=0
--builtin_ntp_servers=127.20.114.212:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:44897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 22:54:40.190142 25540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:40.190382 25540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:40.190456 25540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:40.194383 25540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:40.194469 25540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:40.194500 25540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:40.194535 25540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:40.194561 25540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:40.199311 25540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:44897
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:44897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:40.200340 25540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:40.201786 25540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:40.210029 25546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:40.210027 25545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:40.210364 25540 server_base.cc:1034] running on GCE node
W20260925 22:54:40.210041 25548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:40.210882 25540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:40.211972 25540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:40.213155 25540 hybrid_clock.cc:648] HybridClock initialized: now 1790376880213139 us; error 32 us; skew 500 ppm
I20260925 22:54:40.220351 25540 webserver.cc:460] Webserver started at http://127.20.114.254:32965/ using document root <none> and password file <none>
I20260925 22:54:40.221017 25540 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:40.221122 25540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:40.221359 25540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:40.223441 25540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "e7fa2bff66a5474d9f7b437bb11bb6f9"
format_stamp: "Formatted at 2026-09-25 22:54:40 on dist-test-slave-x1lh"
server_key: "4147e654a6e43fc0b929d8b0ac5a1872"
server_key_iv: "9db46a9625b2d6379c316602bfa4b3e6"
server_key_version: "encryptionkey@0"
I20260925 22:54:40.224012 25540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "e7fa2bff66a5474d9f7b437bb11bb6f9"
format_stamp: "Formatted at 2026-09-25 22:54:40 on dist-test-slave-x1lh"
server_key: "4147e654a6e43fc0b929d8b0ac5a1872"
server_key_iv: "9db46a9625b2d6379c316602bfa4b3e6"
server_key_version: "encryptionkey@0"
I20260925 22:54:40.227646 25540 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 22:54:40.230073 25554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:40.231213 25540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:54:40.231359 25540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "e7fa2bff66a5474d9f7b437bb11bb6f9"
format_stamp: "Formatted at 2026-09-25 22:54:40 on dist-test-slave-x1lh"
server_key: "4147e654a6e43fc0b929d8b0ac5a1872"
server_key_iv: "9db46a9625b2d6379c316602bfa4b3e6"
server_key_version: "encryptionkey@0"
I20260925 22:54:40.231473 25540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:40.235949 25540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:40.236524 25540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:40.236706 25540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:40.243302 25540 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:44897
I20260925 22:54:40.243306 25606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:44897 every 8 connection(s)
I20260925 22:54:40.244244 25540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:54:40.246569 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 25540
I20260925 22:54:40.246690 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:54:40.246994 20939 external_mini_cluster.cc:1428] Setting key 6b6dcc7e8cce15ea9303f29a86703258
I20260925 22:54:40.247668 25607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:40.252584 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:40.248443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35844 (local address 127.20.114.254:44897)
0925 22:54:40.249144 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:40.249159 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:40.249194 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:54:40.249851 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:40.249868 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:40.250213 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:40.250544 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:40.251341 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:40.251345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:40.251357 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:54:40.251366 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:40.251420 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:54:40.251436 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:40.251605 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:40.251675 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 22:54:40.251910 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":133,"threads_started":1}
I20260925 22:54:40.254277 25607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9: Bootstrap starting.
I20260925 22:54:40.256594 25607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:40.257366 25607 log.cc:824] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:40.259282 25607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9: No bootstrap required, opened a new log
I20260925 22:54:40.262789 25607 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fa2bff66a5474d9f7b437bb11bb6f9" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44897 } }
I20260925 22:54:40.263016 25607 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:40.263099 25607 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7fa2bff66a5474d9f7b437bb11bb6f9, State: Initialized, Role: FOLLOWER
I20260925 22:54:40.263552 25607 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fa2bff66a5474d9f7b437bb11bb6f9" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44897 } }
I20260925 22:54:40.263697 25607 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:54:40.263763 25607 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:54:40.263888 25607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:40.264817 25607 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fa2bff66a5474d9f7b437bb11bb6f9" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44897 } }
I20260925 22:54:40.265189 25607 leader_election.cc:302] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7fa2bff66a5474d9f7b437bb11bb6f9; no voters:
I20260925 22:54:40.265491 25607 leader_election.cc:288] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:54:40.265643 25612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:40.265880 25612 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9 [term 1 LEADER]: Becoming Leader. State: Replica: e7fa2bff66a5474d9f7b437bb11bb6f9, State: Running, Role: LEADER
I20260925 22:54:40.266203 25612 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fa2bff66a5474d9f7b437bb11bb6f9" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44897 } }
I20260925 22:54:40.266753 25607 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:40.268059 25613 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7fa2bff66a5474d9f7b437bb11bb6f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fa2bff66a5474d9f7b437bb11bb6f9" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44897 } } }
I20260925 22:54:40.268189 25613 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:40.268059 25614 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7fa2bff66a5474d9f7b437bb11bb6f9. Latest consensus state: current_term: 1 leader_uuid: "e7fa2bff66a5474d9f7b437bb11bb6f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fa2bff66a5474d9f7b437bb11bb6f9" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44897 } } }
I20260925 22:54:40.268597 25614 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:40.268595 25621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:54:40.271708 25621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:54:40.277513 25621 catalog_manager.cc:1347] Generated new cluster ID: 225bda2b0c744c7982a54431129c6837
I20260925 22:54:40.277585 25621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:40.291234 25621 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:54:40.291585 25621 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:40.301901 25621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9: Generated new TSK 0
I20260925 22:54:40.302560 25621 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:54:40.312279 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44897
--builtin_ntp_servers=127.20.114.212:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 22:54:40.422451 25631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:40.422719 25631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:40.422756 25631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:40.426723 25631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:40.426771 25631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:40.426826 25631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:54:40.431409 25631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:40.432430 25631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:40.433828 25631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:40.443189 25636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:40.443197 25637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:40.443293 25631 server_base.cc:1034] running on GCE node
W20260925 22:54:40.443197 25639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:40.443940 25631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:40.444540 25631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:40.445716 25631 hybrid_clock.cc:648] HybridClock initialized: now 1790376880445683 us; error 44 us; skew 500 ppm
I20260925 22:54:40.447615 25631 webserver.cc:460] Webserver started at http://127.20.114.193:43113/ using document root <none> and password file <none>
I20260925 22:54:40.448186 25631 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:40.448277 25631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:40.448511 25631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:40.450542 25631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "23dff056e2e84e928cc62ab944cd1705"
format_stamp: "Formatted at 2026-09-25 22:54:40 on dist-test-slave-x1lh"
server_key: "24a67786c3538b2058bd0068d7570a25"
server_key_iv: "058661f71383b69a617a31e079237343"
server_key_version: "encryptionkey@0"
I20260925 22:54:40.451089 25631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "23dff056e2e84e928cc62ab944cd1705"
format_stamp: "Formatted at 2026-09-25 22:54:40 on dist-test-slave-x1lh"
server_key: "24a67786c3538b2058bd0068d7570a25"
server_key_iv: "058661f71383b69a617a31e079237343"
server_key_version: "encryptionkey@0"
I20260925 22:54:40.454643 25631 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:54:40.457015 25645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:40.458156 25631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:40.458299 25631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "23dff056e2e84e928cc62ab944cd1705"
format_stamp: "Formatted at 2026-09-25 22:54:40 on dist-test-slave-x1lh"
server_key: "24a67786c3538b2058bd0068d7570a25"
server_key_iv: "058661f71383b69a617a31e079237343"
server_key_version: "encryptionkey@0"
I20260925 22:54:40.458415 25631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:40.462878 25631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:40.463552 25631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:40.463748 25631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:40.464401 25631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:40.465616 25631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:40.465685 25631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:40.465749 25631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:40.465780 25631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:40.474983 25631 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:35669
I20260925 22:54:40.475013 25758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:35669 every 8 connection(s)
I20260925 22:54:40.475921 25631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:54:40.478317 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 25631
I20260925 22:54:40.478411 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:54:40.478641 20939 external_mini_cluster.cc:1428] Setting key 0e8c5dace979a10a72972a42fd7d200f
I20260925 22:54:40.480661 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44897
--builtin_ntp_servers=127.20.114.212:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 22:54:40.482370 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:40.477778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:54059 (local address 127.20.114.254:44897)
0925 22:54:40.477921 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:40.477925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:40.479011 (+ 1086us) server_negotiation.cc:407] Connection header received
0925 22:54:40.480512 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:40.480515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:40.480560 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:40.480654 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:40.481940 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:40.481944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:40.481946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:54:40.481956 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:40.481975 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:54:40.481983 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:40.482052 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:40.482235 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 22:54:40.482271 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:54:40.483480 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:40.478135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:44897 (local address 127.20.114.193:54059)
0925 22:54:40.478840 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:40.478890 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:40.480301 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:40.480857 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:40.480870 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:40.480998 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:40.481595 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:54:40.481597 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:40.481639 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:54:40.481746 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:40.482067 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:40.482077 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:40.482083 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:54:40.482278 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 22:54:40.482544 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":133,"threads_started":1}
I20260925 22:54:40.484720 25759 heartbeater.cc:347] Connected to a master server at 127.20.114.254:44897
I20260925 22:54:40.485002 25759 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:40.485401 25759 heartbeater.cc:511] Master 127.20.114.254:44897 requested a full tablet report, sending...
I20260925 22:54:40.486801 25571 ts_manager.cc:194] Registered new tserver with Master: 23dff056e2e84e928cc62ab944cd1705 (127.20.114.193:35669)
W20260925 22:54:40.590934 25762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:40.591216 25762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:40.591290 25762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:40.595312 25762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:40.595378 25762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:40.595465 25762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:54:40.600158 25762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:40.601241 25762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:40.602644 25762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:40.612069 25768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:40.612106 25770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:40.612085 25767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:40.612303 25762 server_base.cc:1034] running on GCE node
I20260925 22:54:40.612793 25762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:40.613440 25762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:40.614621 25762 hybrid_clock.cc:648] HybridClock initialized: now 1790376880614593 us; error 49 us; skew 500 ppm
I20260925 22:54:40.616530 25762 webserver.cc:460] Webserver started at http://127.20.114.194:46431/ using document root <none> and password file <none>
I20260925 22:54:40.617199 25762 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:40.617287 25762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:40.617519 25762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:40.619571 25762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "d2855d191c0041bfa84655a4e65a5dc9"
format_stamp: "Formatted at 2026-09-25 22:54:40 on dist-test-slave-x1lh"
server_key: "d8331492b86e334daaa10aaf2a1804ec"
server_key_iv: "7e54dfadf59a1985992a8b638a4b3e80"
server_key_version: "encryptionkey@0"
I20260925 22:54:40.620151 25762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "d2855d191c0041bfa84655a4e65a5dc9"
format_stamp: "Formatted at 2026-09-25 22:54:40 on dist-test-slave-x1lh"
server_key: "d8331492b86e334daaa10aaf2a1804ec"
server_key_iv: "7e54dfadf59a1985992a8b638a4b3e80"
server_key_version: "encryptionkey@0"
I20260925 22:54:40.623850 25762 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:54:40.626387 25776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:40.627660 25762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:54:40.627792 25762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "d2855d191c0041bfa84655a4e65a5dc9"
format_stamp: "Formatted at 2026-09-25 22:54:40 on dist-test-slave-x1lh"
server_key: "d8331492b86e334daaa10aaf2a1804ec"
server_key_iv: "7e54dfadf59a1985992a8b638a4b3e80"
server_key_version: "encryptionkey@0"
I20260925 22:54:40.627912 25762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:40.632427 25762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:40.633082 25762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:40.633258 25762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:40.633800 25762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:40.634790 25762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:40.634858 25762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:40.634924 25762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:40.634966 25762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:40.643301 25762 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:43573
I20260925 22:54:40.643388 25889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:43573 every 8 connection(s)
I20260925 22:54:40.644237 25762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:54:40.646997 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 25762
I20260925 22:54:40.647130 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:54:40.647439 20939 external_mini_cluster.cc:1428] Setting key f2193eb892441967808b208500322ec6
I20260925 22:54:40.649586 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44897
--builtin_ntp_servers=127.20.114.212:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 22:54:40.650895 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:40.646073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:50491 (local address 127.20.114.254:44897)
0925 22:54:40.646210 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:40.646214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:40.647503 (+ 1289us) server_negotiation.cc:407] Connection header received
0925 22:54:40.648856 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:40.648879 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:40.648920 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:40.648994 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:40.650479 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:40.650481 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:40.650484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:54:40.650495 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:40.650513 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:54:40.650522 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:40.650597 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:40.650761 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 22:54:40.650796 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:54:40.651949 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:40.646502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:44897 (local address 127.20.114.194:50491)
0925 22:54:40.647315 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:40.647372 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:40.648666 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:40.649169 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:40.649177 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:40.649289 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:40.650100 (+ 811us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:54:40.650104 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:40.650188 (+ 84us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:54:40.650322 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:40.650613 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:40.650623 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:40.650632 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 22:54:40.650805 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 22:54:40.651232 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":108,"threads_started":1}
I20260925 22:54:40.653106 25890 heartbeater.cc:347] Connected to a master server at 127.20.114.254:44897
I20260925 22:54:40.653337 25890 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:40.653748 25890 heartbeater.cc:511] Master 127.20.114.254:44897 requested a full tablet report, sending...
I20260925 22:54:40.654762 25571 ts_manager.cc:194] Registered new tserver with Master: d2855d191c0041bfa84655a4e65a5dc9 (127.20.114.194:43573)
W20260925 22:54:40.759527 25893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:40.759827 25893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:40.759907 25893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:40.764036 25893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:40.764120 25893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:40.764210 25893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:54:40.769279 25893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:40.770359 25893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:40.771725 25893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:40.781399 25901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:40.781443 25899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:40.781435 25898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:40.781752 25893 server_base.cc:1034] running on GCE node
I20260925 22:54:40.782197 25893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:40.782876 25893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:40.784046 25893 hybrid_clock.cc:648] HybridClock initialized: now 1790376880784026 us; error 32 us; skew 500 ppm
I20260925 22:54:40.785972 25893 webserver.cc:460] Webserver started at http://127.20.114.195:37863/ using document root <none> and password file <none>
I20260925 22:54:40.786547 25893 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:40.786608 25893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:40.786863 25893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:40.788956 25893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "9b05d0e177a64e46ad00cf5b15d20193"
format_stamp: "Formatted at 2026-09-25 22:54:40 on dist-test-slave-x1lh"
server_key: "ccf4d39c1dbaa1685abedea01d6a1e90"
server_key_iv: "e97fc00a08af49c6b6ac8b5edc1ea098"
server_key_version: "encryptionkey@0"
I20260925 22:54:40.789533 25893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "9b05d0e177a64e46ad00cf5b15d20193"
format_stamp: "Formatted at 2026-09-25 22:54:40 on dist-test-slave-x1lh"
server_key: "ccf4d39c1dbaa1685abedea01d6a1e90"
server_key_iv: "e97fc00a08af49c6b6ac8b5edc1ea098"
server_key_version: "encryptionkey@0"
I20260925 22:54:40.793180 25893 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 22:54:40.795751 25907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:40.797027 25893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:40.797163 25893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "9b05d0e177a64e46ad00cf5b15d20193"
format_stamp: "Formatted at 2026-09-25 22:54:40 on dist-test-slave-x1lh"
server_key: "ccf4d39c1dbaa1685abedea01d6a1e90"
server_key_iv: "e97fc00a08af49c6b6ac8b5edc1ea098"
server_key_version: "encryptionkey@0"
I20260925 22:54:40.797279 25893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:40.801855 25893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:40.802493 25893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:40.802690 25893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:40.803278 25893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:40.804210 25893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:40.804255 25893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:40.804317 25893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:40.804352 25893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:40.812727 25893 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:41293
I20260925 22:54:40.812747 26020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:41293 every 8 connection(s)
I20260925 22:54:40.813643 25893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:54:40.815850 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 25893
I20260925 22:54:40.815984 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:54:40.816265 20939 external_mini_cluster.cc:1428] Setting key e6def9b637908b427094f48a374034ba
I20260925 22:54:40.819638 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:40.815414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:37231 (local address 127.20.114.254:44897)
0925 22:54:40.815544 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:40.815548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:40.816744 (+ 1196us) server_negotiation.cc:407] Connection header received
0925 22:54:40.818125 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:40.818128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:40.818167 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:40.818240 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:40.819318 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:40.819321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:40.819323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:54:40.819334 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:40.819352 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:54:40.819360 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:40.819426 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:40.819504 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 22:54:40.819537 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:54:40.820540 26023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:40.815836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:44897 (local address 127.20.114.195:37231)
0925 22:54:40.816573 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:40.816623 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:40.817922 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:40.818374 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:40.818381 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:40.818471 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:40.819084 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:54:40.819086 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:40.819125 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:54:40.819194 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:40.819417 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:40.819421 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:40.819425 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:54:40.819542 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 22:54:40.819777 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":159,"threads_started":1}
I20260925 22:54:40.821651 26021 heartbeater.cc:347] Connected to a master server at 127.20.114.254:44897
I20260925 22:54:40.821885 26021 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:40.822299 26021 heartbeater.cc:511] Master 127.20.114.254:44897 requested a full tablet report, sending...
I20260925 22:54:40.823259 25571 ts_manager.cc:194] Registered new tserver with Master: 9b05d0e177a64e46ad00cf5b15d20193 (127.20.114.195:41293)
I20260925 22:54:40.830178 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:54:40.834182 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:40.832339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35856 (local address 127.20.114.254:44897)
0925 22:54:40.832500 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:40.832507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:40.832706 (+ 199us) server_negotiation.cc:407] Connection header received
0925 22:54:40.832910 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:40.832913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:40.832963 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:40.833075 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:40.833759 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:40.833763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:40.833765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:54:40.833777 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:40.833798 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:54:40.833814 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:40.833904 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:40.833989 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 22:54:40.834031 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:54:40.836997 25571 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:54:40.839994 25571 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:54:40.851727 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:40.848644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:35669 (local address 127.0.0.1:59060)
0925 22:54:40.849365 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:40.849378 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:40.849454 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:40.850019 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:40.850026 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:40.850101 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:40.850681 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:54:40.850683 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:40.850703 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:54:40.850771 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:40.851475 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:40.851481 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:40.851487 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:54:40.851556 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 22:54:40.851579 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":103,"threads_started":1}
I20260925 22:54:40.851876 26033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:40.848388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46268 (local address 127.20.114.195:41293)
0925 22:54:40.849671 (+ 1283us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:40.849682 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:40.849703 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:54:40.849754 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:40.849758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:40.849890 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:40.850028 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:40.851210 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:40.851217 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:40.851229 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:54:40.851240 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:40.851283 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:54:40.851305 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:40.851410 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:40.851658 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 22:54:40.851721 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1154,"thread_start_us":159,"threads_started":1}
I20260925 22:54:40.852010 26037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:40.849069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59060 (local address 127.20.114.193:35669)
0925 22:54:40.849548 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:40.849563 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:40.849587 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:54:40.849645 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:40.849652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:40.849859 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:40.850002 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:40.851315 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:40.851320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:40.851328 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:54:40.851339 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:40.851372 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:54:40.851397 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:40.851493 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:40.851593 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 22:54:40.851668 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":97,"threads_started":1}
I20260925 22:54:40.852124 26035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:40.848431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:43573 (local address 127.0.0.1:44354)
0925 22:54:40.849208 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:40.849258 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:40.849363 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:40.850188 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:40.850191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:40.850207 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:40.850845 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:54:40.850851 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:40.850865 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:54:40.850895 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:40.851430 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:40.851434 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:40.851439 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:54:40.851496 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 22:54:40.851581 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":124,"threads_started":1}
I20260925 22:54:40.852324 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:40.848211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:41293 (local address 127.0.0.1:46268)
0925 22:54:40.849544 (+ 1333us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:40.849554 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:40.849603 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:40.850696 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:40.850698 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:40.850712 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:40.851066 (+ 354us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:54:40.851067 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:40.851075 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:54:40.851094 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:40.851394 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:40.851403 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:40.851408 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:54:40.851519 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 22:54:40.851574 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":93,"threads_started":1}
I20260925 22:54:40.852810 26036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:40.848954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44354 (local address 127.20.114.194:43573)
0925 22:54:40.849821 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:40.849830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:40.849850 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:54:40.849898 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:40.849907 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:40.850048 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:40.850413 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:40.851230 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:40.851236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:40.851246 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:54:40.851256 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:40.851297 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:54:40.851319 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:40.852566 (+ 1247us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:40.852610 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 22:54:40.852670 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":137,"threads_started":1}
I20260925 22:54:40.854794 25943 tablet_service.cc:1467] Processing CreateTablet for tablet 01bb7432d8874ab0baf0f30589878c1e (DEFAULT_TABLE table=test-table [id=7f2ca13817c54962b929479b5382c0e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:40.854849 25693 tablet_service.cc:1467] Processing CreateTablet for tablet 01bb7432d8874ab0baf0f30589878c1e (DEFAULT_TABLE table=test-table [id=7f2ca13817c54962b929479b5382c0e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:40.855381 25824 tablet_service.cc:1467] Processing CreateTablet for tablet 01bb7432d8874ab0baf0f30589878c1e (DEFAULT_TABLE table=test-table [id=7f2ca13817c54962b929479b5382c0e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:40.855878 25943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01bb7432d8874ab0baf0f30589878c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:40.855877 25693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01bb7432d8874ab0baf0f30589878c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:40.856225 25824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01bb7432d8874ab0baf0f30589878c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:40.861779 26038 tablet_bootstrap.cc:492] T 01bb7432d8874ab0baf0f30589878c1e P 9b05d0e177a64e46ad00cf5b15d20193: Bootstrap starting.
I20260925 22:54:40.863390 26039 tablet_bootstrap.cc:492] T 01bb7432d8874ab0baf0f30589878c1e P d2855d191c0041bfa84655a4e65a5dc9: Bootstrap starting.
I20260925 22:54:40.863390 26040 tablet_bootstrap.cc:492] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705: Bootstrap starting.
I20260925 22:54:40.864269 26038 tablet_bootstrap.cc:654] T 01bb7432d8874ab0baf0f30589878c1e P 9b05d0e177a64e46ad00cf5b15d20193: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:40.865057 26038 log.cc:824] T 01bb7432d8874ab0baf0f30589878c1e P 9b05d0e177a64e46ad00cf5b15d20193: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:40.865484 26040 tablet_bootstrap.cc:654] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:40.865484 26039 tablet_bootstrap.cc:654] T 01bb7432d8874ab0baf0f30589878c1e P d2855d191c0041bfa84655a4e65a5dc9: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:40.866304 26040 log.cc:824] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:40.866304 26039 log.cc:824] T 01bb7432d8874ab0baf0f30589878c1e P d2855d191c0041bfa84655a4e65a5dc9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:40.866935 26038 tablet_bootstrap.cc:492] T 01bb7432d8874ab0baf0f30589878c1e P 9b05d0e177a64e46ad00cf5b15d20193: No bootstrap required, opened a new log
I20260925 22:54:40.867177 26038 ts_tablet_manager.cc:1397] T 01bb7432d8874ab0baf0f30589878c1e P 9b05d0e177a64e46ad00cf5b15d20193: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:54:40.868101 26039 tablet_bootstrap.cc:492] T 01bb7432d8874ab0baf0f30589878c1e P d2855d191c0041bfa84655a4e65a5dc9: No bootstrap required, opened a new log
I20260925 22:54:40.868285 26039 ts_tablet_manager.cc:1397] T 01bb7432d8874ab0baf0f30589878c1e P d2855d191c0041bfa84655a4e65a5dc9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:54:40.870095 26040 tablet_bootstrap.cc:492] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705: No bootstrap required, opened a new log
I20260925 22:54:40.870267 26040 ts_tablet_manager.cc:1397] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260925 22:54:40.871196 26038 raft_consensus.cc:348] T 01bb7432d8874ab0baf0f30589878c1e P 9b05d0e177a64e46ad00cf5b15d20193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b05d0e177a64e46ad00cf5b15d20193" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 41293 } } peers { permanent_uuid: "d2855d191c0041bfa84655a4e65a5dc9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43573 } } peers { permanent_uuid: "23dff056e2e84e928cc62ab944cd1705" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35669 } }
I20260925 22:54:40.871425 26038 raft_consensus.cc:374] T 01bb7432d8874ab0baf0f30589878c1e P 9b05d0e177a64e46ad00cf5b15d20193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:40.871476 26038 raft_consensus.cc:729] T 01bb7432d8874ab0baf0f30589878c1e P 9b05d0e177a64e46ad00cf5b15d20193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b05d0e177a64e46ad00cf5b15d20193, State: Initialized, Role: FOLLOWER
I20260925 22:54:40.871927 26038 consensus_queue.cc:260] T 01bb7432d8874ab0baf0f30589878c1e P 9b05d0e177a64e46ad00cf5b15d20193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b05d0e177a64e46ad00cf5b15d20193" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 41293 } } peers { permanent_uuid: "d2855d191c0041bfa84655a4e65a5dc9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43573 } } peers { permanent_uuid: "23dff056e2e84e928cc62ab944cd1705" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35669 } }
I20260925 22:54:40.872534 26021 heartbeater.cc:503] Master 127.20.114.254:44897 was elected leader, sending a full tablet report...
I20260925 22:54:40.872574 26039 raft_consensus.cc:348] T 01bb7432d8874ab0baf0f30589878c1e P d2855d191c0041bfa84655a4e65a5dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b05d0e177a64e46ad00cf5b15d20193" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 41293 } } peers { permanent_uuid: "d2855d191c0041bfa84655a4e65a5dc9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43573 } } peers { permanent_uuid: "23dff056e2e84e928cc62ab944cd1705" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35669 } }
I20260925 22:54:40.872809 26039 raft_consensus.cc:374] T 01bb7432d8874ab0baf0f30589878c1e P d2855d191c0041bfa84655a4e65a5dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:40.872853 26039 raft_consensus.cc:729] T 01bb7432d8874ab0baf0f30589878c1e P d2855d191c0041bfa84655a4e65a5dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2855d191c0041bfa84655a4e65a5dc9, State: Initialized, Role: FOLLOWER
I20260925 22:54:40.872922 26038 ts_tablet_manager.cc:1428] T 01bb7432d8874ab0baf0f30589878c1e P 9b05d0e177a64e46ad00cf5b15d20193: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:54:40.873337 26039 consensus_queue.cc:260] T 01bb7432d8874ab0baf0f30589878c1e P d2855d191c0041bfa84655a4e65a5dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b05d0e177a64e46ad00cf5b15d20193" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 41293 } } peers { permanent_uuid: "d2855d191c0041bfa84655a4e65a5dc9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43573 } } peers { permanent_uuid: "23dff056e2e84e928cc62ab944cd1705" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35669 } }
I20260925 22:54:40.874011 25890 heartbeater.cc:503] Master 127.20.114.254:44897 was elected leader, sending a full tablet report...
I20260925 22:54:40.874473 26039 ts_tablet_manager.cc:1428] T 01bb7432d8874ab0baf0f30589878c1e P d2855d191c0041bfa84655a4e65a5dc9: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260925 22:54:40.874728 26040 raft_consensus.cc:348] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b05d0e177a64e46ad00cf5b15d20193" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 41293 } } peers { permanent_uuid: "d2855d191c0041bfa84655a4e65a5dc9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43573 } } peers { permanent_uuid: "23dff056e2e84e928cc62ab944cd1705" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35669 } }
I20260925 22:54:40.874994 26040 raft_consensus.cc:374] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:40.875053 26040 raft_consensus.cc:729] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23dff056e2e84e928cc62ab944cd1705, State: Initialized, Role: FOLLOWER
I20260925 22:54:40.875434 26040 consensus_queue.cc:260] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b05d0e177a64e46ad00cf5b15d20193" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 41293 } } peers { permanent_uuid: "d2855d191c0041bfa84655a4e65a5dc9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43573 } } peers { permanent_uuid: "23dff056e2e84e928cc62ab944cd1705" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35669 } }
I20260925 22:54:40.876166 25759 heartbeater.cc:503] Master 127.20.114.254:44897 was elected leader, sending a full tablet report...
I20260925 22:54:40.876356 26040 ts_tablet_manager.cc:1428] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260925 22:54:40.895238 25891 tablet.cc:2369] T 01bb7432d8874ab0baf0f30589878c1e P d2855d191c0041bfa84655a4e65a5dc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:54:40.927749 26046 raft_consensus.cc:482] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:54:40.927964 26046 raft_consensus.cc:504] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b05d0e177a64e46ad00cf5b15d20193" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 41293 } } peers { permanent_uuid: "d2855d191c0041bfa84655a4e65a5dc9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43573 } } peers { permanent_uuid: "23dff056e2e84e928cc62ab944cd1705" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35669 } }
I20260925 22:54:40.929183 26046 leader_election.cc:288] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b05d0e177a64e46ad00cf5b15d20193 (127.20.114.195:41293), d2855d191c0041bfa84655a4e65a5dc9 (127.20.114.194:43573)
I20260925 22:54:40.931093 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:40.929360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:41293 (local address 127.20.114.193:52557)
0925 22:54:40.929497 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:40.929513 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:40.929612 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:40.929905 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:40.929907 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:40.929925 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:40.930484 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:54:40.930486 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:40.930495 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:54:40.930544 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:40.930820 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:40.930823 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:40.930826 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 22:54:40.930905 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 22:54:40.930951 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 22:54:40.931149 26033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:40.929473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:52557 (local address 127.20.114.195:41293)
0925 22:54:40.929635 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:40.929640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:40.929683 (+ 43us) server_negotiation.cc:407] Connection header received
0925 22:54:40.929732 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:40.929735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:40.929773 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:40.929859 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:40.930689 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:40.930692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:40.930695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:54:40.930707 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:40.930729 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:54:40.930739 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:40.930832 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:40.930961 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 22:54:40.931007 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:54:40.931502 26036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:40.929670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:38261 (local address 127.20.114.194:43573)
0925 22:54:40.929775 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:40.929779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:40.929904 (+ 125us) server_negotiation.cc:407] Connection header received
0925 22:54:40.929982 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:40.929986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:40.930024 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:40.930111 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:40.931016 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:40.931019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:40.931022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:54:40.931033 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:40.931055 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:54:40.931065 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:40.931169 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:40.931338 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 22:54:40.931381 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 22:54:40.931514 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:40.929428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:43573 (local address 127.20.114.193:38261)
0925 22:54:40.929812 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:40.929827 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:40.929904 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:40.930161 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:40.930166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:40.930182 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:40.930894 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:54:40.930896 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:40.930906 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:54:40.930928 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:40.931196 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:40.931200 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:40.931203 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 22:54:40.931326 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 22:54:40.931363 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":129,"threads_started":1}
I20260925 22:54:40.931730 25975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01bb7432d8874ab0baf0f30589878c1e" candidate_uuid: "23dff056e2e84e928cc62ab944cd1705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b05d0e177a64e46ad00cf5b15d20193" is_pre_election: true
I20260925 22:54:40.932029 25975 raft_consensus.cc:2454] T 01bb7432d8874ab0baf0f30589878c1e P 9b05d0e177a64e46ad00cf5b15d20193 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23dff056e2e84e928cc62ab944cd1705 in term 0.
I20260925 22:54:40.932175 25844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01bb7432d8874ab0baf0f30589878c1e" candidate_uuid: "23dff056e2e84e928cc62ab944cd1705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2855d191c0041bfa84655a4e65a5dc9" is_pre_election: true
I20260925 22:54:40.932464 25844 raft_consensus.cc:2454] T 01bb7432d8874ab0baf0f30589878c1e P d2855d191c0041bfa84655a4e65a5dc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23dff056e2e84e928cc62ab944cd1705 in term 0.
I20260925 22:54:40.932483 25647 leader_election.cc:302] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23dff056e2e84e928cc62ab944cd1705, 9b05d0e177a64e46ad00cf5b15d20193; no voters:
I20260925 22:54:40.932776 26046 raft_consensus.cc:2781] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:54:40.932880 26046 raft_consensus.cc:482] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:54:40.932940 26046 raft_consensus.cc:3037] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:40.933957 26046 raft_consensus.cc:504] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b05d0e177a64e46ad00cf5b15d20193" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 41293 } } peers { permanent_uuid: "d2855d191c0041bfa84655a4e65a5dc9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43573 } } peers { permanent_uuid: "23dff056e2e84e928cc62ab944cd1705" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35669 } }
I20260925 22:54:40.934329 26046 leader_election.cc:288] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705 [CANDIDATE]: Term 1 election: Requested vote from peers 9b05d0e177a64e46ad00cf5b15d20193 (127.20.114.195:41293), d2855d191c0041bfa84655a4e65a5dc9 (127.20.114.194:43573)
I20260925 22:54:40.934626 25975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01bb7432d8874ab0baf0f30589878c1e" candidate_uuid: "23dff056e2e84e928cc62ab944cd1705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b05d0e177a64e46ad00cf5b15d20193"
I20260925 22:54:40.934695 25844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01bb7432d8874ab0baf0f30589878c1e" candidate_uuid: "23dff056e2e84e928cc62ab944cd1705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2855d191c0041bfa84655a4e65a5dc9"
I20260925 22:54:40.934759 25975 raft_consensus.cc:3037] T 01bb7432d8874ab0baf0f30589878c1e P 9b05d0e177a64e46ad00cf5b15d20193 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:40.934811 25844 raft_consensus.cc:3037] T 01bb7432d8874ab0baf0f30589878c1e P d2855d191c0041bfa84655a4e65a5dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:40.935818 25975 raft_consensus.cc:2454] T 01bb7432d8874ab0baf0f30589878c1e P 9b05d0e177a64e46ad00cf5b15d20193 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23dff056e2e84e928cc62ab944cd1705 in term 1.
I20260925 22:54:40.936051 25844 raft_consensus.cc:2454] T 01bb7432d8874ab0baf0f30589878c1e P d2855d191c0041bfa84655a4e65a5dc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23dff056e2e84e928cc62ab944cd1705 in term 1.
I20260925 22:54:40.936085 25647 leader_election.cc:302] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23dff056e2e84e928cc62ab944cd1705, 9b05d0e177a64e46ad00cf5b15d20193; no voters:
I20260925 22:54:40.936287 26046 raft_consensus.cc:2781] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:40.936502 26046 raft_consensus.cc:686] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705 [term 1 LEADER]: Becoming Leader. State: Replica: 23dff056e2e84e928cc62ab944cd1705, State: Running, Role: LEADER
I20260925 22:54:40.936807 26046 consensus_queue.cc:237] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b05d0e177a64e46ad00cf5b15d20193" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 41293 } } peers { permanent_uuid: "d2855d191c0041bfa84655a4e65a5dc9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43573 } } peers { permanent_uuid: "23dff056e2e84e928cc62ab944cd1705" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35669 } }
I20260925 22:54:40.939945 25570 catalog_manager.cc:5495] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23dff056e2e84e928cc62ab944cd1705 (127.20.114.193). New cstate: current_term: 1 leader_uuid: "23dff056e2e84e928cc62ab944cd1705" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b05d0e177a64e46ad00cf5b15d20193" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 41293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2855d191c0041bfa84655a4e65a5dc9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23dff056e2e84e928cc62ab944cd1705" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 35669 } health_report { overall_health: HEALTHY } } }
I20260925 22:54:40.948547 26037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:40.947216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59070 (local address 127.20.114.193:35669)
0925 22:54:40.947335 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:40.947338 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:40.947350 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:54:40.947473 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:40.947475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:40.947525 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:40.947585 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:40.948240 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:40.948242 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:40.948244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:54:40.948252 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:40.948265 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:54:40.948271 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:40.948329 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:40.948370 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 22:54:40.948398 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:54:40.953836 25844 raft_consensus.cc:1261] T 01bb7432d8874ab0baf0f30589878c1e P d2855d191c0041bfa84655a4e65a5dc9 [term 1 FOLLOWER]: Refusing update from remote peer 23dff056e2e84e928cc62ab944cd1705: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:40.953836 25975 raft_consensus.cc:1261] T 01bb7432d8874ab0baf0f30589878c1e P 9b05d0e177a64e46ad00cf5b15d20193 [term 1 FOLLOWER]: Refusing update from remote peer 23dff056e2e84e928cc62ab944cd1705: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:40.954499 26048 consensus_queue.cc:1035] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2855d191c0041bfa84655a4e65a5dc9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:40.954667 26046 consensus_queue.cc:1035] T 01bb7432d8874ab0baf0f30589878c1e P 23dff056e2e84e928cc62ab944cd1705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b05d0e177a64e46ad00cf5b15d20193" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 41293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:40.962604 26052 mvcc.cc:205] Tried to move back new op lower bound from 7333383704380297216 to 7333383704319483904. Current Snapshot: MvccSnapshot[applied={T|T < 7333383704380297216}]
I20260925 22:54:40.962815 26053 mvcc.cc:205] Tried to move back new op lower bound from 7333383704380297216 to 7333383704319483904. Current Snapshot: MvccSnapshot[applied={T|T < 7333383704380297216}]
I20260925 22:54:40.969668 25569 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:40.969830 25569 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:40.971920 25569 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7fa2bff66a5474d9f7b437bb11bb6f9: Sending DeleteTablet for 3 replicas of tablet 01bb7432d8874ab0baf0f30589878c1e
I20260925 22:54:40.972612 25943 tablet_service.cc:1514] Processing DeleteTablet for tablet 01bb7432d8874ab0baf0f30589878c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:40 UTC) from {username='slave'} at 127.0.0.1:46268
I20260925 22:54:40.972750 25824 tablet_service.cc:1514] Processing DeleteTablet for tablet 01bb7432d8874ab0baf0f30589878c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:40 UTC) from {username='slave'} at 127.0.0.1:44354
I20260925 22:54:40.972750 25693 tablet_service.cc:1514] Processing DeleteTablet for tablet 01bb7432d8874ab0baf0f30589878c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:40 UTC) from {username='slave'} at 127.0.0.1:59060
I20260925 22:54:40.973373 26060 tablet_replica.cc:330] T 01bb7432d8874ab0baf0f30589878c1e P 9b05d0e177a64e46ad00cf5b15d20193: stopping tablet replica
I20260925 22:54:40.973676 26060 raft_consensus.cc:2229] T 01bb7432d8874ab0baf0f30589878c1e P 9b05d0e177a64e46ad00cf5b15d20193 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:40.973701 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 25631
I20260925 22:54:40.974529 26060 raft_consensus.cc:2258] T 01bb7432d8874ab0baf0f30589878c1e P 9b05d0e177a64e46ad00cf5b15d20193 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:40.974920 26061 tablet_replica.cc:330] T 01bb7432d8874ab0baf0f30589878c1e P d2855d191c0041bfa84655a4e65a5dc9: stopping tablet replica
I20260925 22:54:40.975198 26061 raft_consensus.cc:2229] T 01bb7432d8874ab0baf0f30589878c1e P d2855d191c0041bfa84655a4e65a5dc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:40.975497 26061 raft_consensus.cc:2258] T 01bb7432d8874ab0baf0f30589878c1e P d2855d191c0041bfa84655a4e65a5dc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:40.976208 26060 ts_tablet_manager.cc:1905] T 01bb7432d8874ab0baf0f30589878c1e P 9b05d0e177a64e46ad00cf5b15d20193: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:40.976770 26061 ts_tablet_manager.cc:1905] T 01bb7432d8874ab0baf0f30589878c1e P d2855d191c0041bfa84655a4e65a5dc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:40.979043 26061 ts_tablet_manager.cc:1918] T 01bb7432d8874ab0baf0f30589878c1e P d2855d191c0041bfa84655a4e65a5dc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:54:40.979161 26061 log.cc:1196] T 01bb7432d8874ab0baf0f30589878c1e P d2855d191c0041bfa84655a4e65a5dc9: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/01bb7432d8874ab0baf0f30589878c1e
I20260925 22:54:40.979480 26061 ts_tablet_manager.cc:1939] T 01bb7432d8874ab0baf0f30589878c1e P d2855d191c0041bfa84655a4e65a5dc9: Deleting consensus metadata
I20260925 22:54:40.979694 26060 ts_tablet_manager.cc:1918] T 01bb7432d8874ab0baf0f30589878c1e P 9b05d0e177a64e46ad00cf5b15d20193: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:54:40.979794 26060 log.cc:1196] T 01bb7432d8874ab0baf0f30589878c1e P 9b05d0e177a64e46ad00cf5b15d20193: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/01bb7432d8874ab0baf0f30589878c1e
I20260925 22:54:40.980113 26060 ts_tablet_manager.cc:1939] T 01bb7432d8874ab0baf0f30589878c1e P 9b05d0e177a64e46ad00cf5b15d20193: Deleting consensus metadata
I20260925 22:54:40.980669 25558 catalog_manager.cc:4841] TS d2855d191c0041bfa84655a4e65a5dc9 (127.20.114.194:43573): tablet 01bb7432d8874ab0baf0f30589878c1e (table test-table [id=7f2ca13817c54962b929479b5382c0e7]) successfully deleted
I20260925 22:54:40.981276 25556 catalog_manager.cc:4841] TS 9b05d0e177a64e46ad00cf5b15d20193 (127.20.114.195:41293): tablet 01bb7432d8874ab0baf0f30589878c1e (table test-table [id=7f2ca13817c54962b929479b5382c0e7]) successfully deleted
W20260925 22:54:40.981941 25555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.114.193:35669 (error 108)
I20260925 22:54:40.982105 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 25762
I20260925 22:54:40.983392 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:40.983017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:35669 (local address 127.0.0.1:59084)
0925 22:54:40.983213 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:40.983289 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.193:35669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260925 22:54:40.983559 25555 catalog_manager.cc:4584] TS 23dff056e2e84e928cc62ab944cd1705 (127.20.114.193:35669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01bb7432d8874ab0baf0f30589878c1e: Network error: Client connection negotiation failed: client connection to 127.20.114.193:35669: connect: Connection refused (error 111)
I20260925 22:54:40.987903 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 25893
I20260925 22:54:40.993093 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 25540
2026-09-25T22:54:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (940 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 22:54:41.014238 20939 mini_oidc.cc:152] Starting JWKS server
I20260925 22:54:41.015815 20939 webserver.cc:460] Webserver started at https://127.0.0.1:42697/ using document root <none> and password file <none>
I20260925 22:54:41.015949 20939 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 22:54:41.016173 20939 webserver.cc:460] Webserver started at http://127.0.0.1:41277/ using document root <none> and password file <none>
2026-09-25T22:54:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:54:41Z Disabled control of system clock
I20260925 22:54:41.025308 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:41967
--webserver_interface=127.20.114.254
--webserver_port=0
--builtin_ntp_servers=127.20.114.212:35883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:41967
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42697/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376856092611-20939-0/testchainca.cert with env {}
W20260925 22:54:41.133291 26069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:41.133541 26069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:41.133580 26069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:41.137517 26069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:41.137568 26069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:41.137583 26069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:41.137600 26069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:41.137622 26069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:54:41.137638 26069 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:54:41.137651 26069 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42697/jwks.json
I20260925 22:54:41.142889 26069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:41967
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:41967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42697/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376856092611-20939-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:41.143841 26069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:41.144776 26069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:41.153139 26077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:41.153225 26069 server_base.cc:1034] running on GCE node
W20260925 22:54:41.153148 26074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:41.153148 26075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:41.153951 26069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:41.155224 26069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:41.156394 26069 hybrid_clock.cc:648] HybridClock initialized: now 1790376881156366 us; error 47 us; skew 500 ppm
I20260925 22:54:41.158675 26069 webserver.cc:460] Webserver started at http://127.20.114.254:40007/ using document root <none> and password file <none>
I20260925 22:54:41.159261 26069 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:41.159323 26069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:41.159543 26069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:41.161660 26069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "0502e45b9be34684ab88e3914d01131d"
format_stamp: "Formatted at 2026-09-25 22:54:41 on dist-test-slave-x1lh"
server_key: "e9efa83494e67f5775e9736524559f2a"
server_key_iv: "e5034a05d308cc76b369876b7416d14f"
server_key_version: "encryptionkey@0"
I20260925 22:54:41.162220 26069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "0502e45b9be34684ab88e3914d01131d"
format_stamp: "Formatted at 2026-09-25 22:54:41 on dist-test-slave-x1lh"
server_key: "e9efa83494e67f5775e9736524559f2a"
server_key_iv: "e5034a05d308cc76b369876b7416d14f"
server_key_version: "encryptionkey@0"
I20260925 22:54:41.165803 26069 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:54:41.168176 26083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:41.169459 26069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:41.169598 26069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "0502e45b9be34684ab88e3914d01131d"
format_stamp: "Formatted at 2026-09-25 22:54:41 on dist-test-slave-x1lh"
server_key: "e9efa83494e67f5775e9736524559f2a"
server_key_iv: "e5034a05d308cc76b369876b7416d14f"
server_key_version: "encryptionkey@0"
I20260925 22:54:41.169708 26069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:41.186635 26069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:41.187300 26069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:41.187489 26069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:41.194195 26069 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:41967
I20260925 22:54:41.194201 26135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:41967 every 8 connection(s)
I20260925 22:54:41.195107 26069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:54:41.198295 26136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:41.200893 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 26069
I20260925 22:54:41.201020 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:54:41.201359 20939 external_mini_cluster.cc:1428] Setting key c3c5821ebecc557d5fc3594f0e7fb500
I20260925 22:54:41.204361 26136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0502e45b9be34684ab88e3914d01131d: Bootstrap starting.
I20260925 22:54:41.206585 26136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0502e45b9be34684ab88e3914d01131d: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:41.207333 26136 log.cc:824] T 00000000000000000000000000000000 P 0502e45b9be34684ab88e3914d01131d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:41.209510 26136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0502e45b9be34684ab88e3914d01131d: No bootstrap required, opened a new log
I20260925 22:54:41.210639 26139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:41.202696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44590 (local address 127.20.114.254:41967)
0925 22:54:41.203312 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:41.203326 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:41.203359 (+ 33us) server_negotiation.cc:407] Connection header received
0925 22:54:41.203969 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:41.203985 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:41.204377 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:41.204761 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:41.205230 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:41.206168 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:41.207113 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:41.207642 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:41.207818 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:41.208621 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:41.208627 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:41.208642 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:54:41.209153 (+ 511us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:41.209222 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:54:41.209249 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:41.209343 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:41.209490 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 22:54:41.209753 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":122,"threads_started":1}
I20260925 22:54:41.213356 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:54:41.213673 26136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0502e45b9be34684ab88e3914d01131d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0502e45b9be34684ab88e3914d01131d" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 41967 } }
I20260925 22:54:41.213907 26136 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0502e45b9be34684ab88e3914d01131d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:41.213982 26136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0502e45b9be34684ab88e3914d01131d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0502e45b9be34684ab88e3914d01131d, State: Initialized, Role: FOLLOWER
I20260925 22:54:41.214447 26136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0502e45b9be34684ab88e3914d01131d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0502e45b9be34684ab88e3914d01131d" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 41967 } }
I20260925 22:54:41.214614 26136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0502e45b9be34684ab88e3914d01131d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:54:41.214700 26136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0502e45b9be34684ab88e3914d01131d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:54:41.214794 26136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0502e45b9be34684ab88e3914d01131d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:41.215708 26136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0502e45b9be34684ab88e3914d01131d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0502e45b9be34684ab88e3914d01131d" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 41967 } }
I20260925 22:54:41.216058 26136 leader_election.cc:302] T 00000000000000000000000000000000 P 0502e45b9be34684ab88e3914d01131d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0502e45b9be34684ab88e3914d01131d; no voters:
I20260925 22:54:41.216370 26136 leader_election.cc:288] T 00000000000000000000000000000000 P 0502e45b9be34684ab88e3914d01131d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:54:41.216511 26141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0502e45b9be34684ab88e3914d01131d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:41.216742 26141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0502e45b9be34684ab88e3914d01131d [term 1 LEADER]: Becoming Leader. State: Replica: 0502e45b9be34684ab88e3914d01131d, State: Running, Role: LEADER
I20260925 22:54:41.217059 26141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0502e45b9be34684ab88e3914d01131d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0502e45b9be34684ab88e3914d01131d" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 41967 } }
I20260925 22:54:41.217629 26136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0502e45b9be34684ab88e3914d01131d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:41.218750 26142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0502e45b9be34684ab88e3914d01131d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0502e45b9be34684ab88e3914d01131d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0502e45b9be34684ab88e3914d01131d" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 41967 } } }
I20260925 22:54:41.218885 26142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0502e45b9be34684ab88e3914d01131d [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:41.219151 26143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0502e45b9be34684ab88e3914d01131d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0502e45b9be34684ab88e3914d01131d. Latest consensus state: current_term: 1 leader_uuid: "0502e45b9be34684ab88e3914d01131d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0502e45b9be34684ab88e3914d01131d" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 41967 } } }
I20260925 22:54:41.219261 26143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0502e45b9be34684ab88e3914d01131d [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:41.219235 26150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:54:41.222085 26150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:54:41.227082 26150 catalog_manager.cc:1347] Generated new cluster ID: 1b3c1441481941a9b8aed2d1deb08d52
I20260925 22:54:41.227146 26150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:41.240932 26150 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:54:41.242094 26150 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:41.248309 26150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0502e45b9be34684ab88e3914d01131d: Generated new TSK 0
I20260925 22:54:41.249172 26150 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 22:54:41.282301 26166 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 22:54:41.282424 26166 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 22:54:41.283075 26139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:41.261064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44596 (local address 127.20.114.254:41967)
0925 22:54:41.261204 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:41.261209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:41.261440 (+ 231us) server_negotiation.cc:407] Connection header received
0925 22:54:41.261675 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:41.261678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:41.261724 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:41.261800 (+ 76us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:54:41.262061 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:41.262573 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:41.263781 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:41.263988 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:41.264060 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:41.264408 (+ 348us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:54:41.282628 (+ 18220us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 22:54:41.282922 (+ 294us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44596: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 22:54:41.283142 26139 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44596: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 22:54:41.284276 26165 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:41.260986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:41967 (local address 127.0.0.1:44596)
0925 22:54:41.261349 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:41.261366 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:41.261495 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:41.261841 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:41.261844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:41.261853 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:54:41.261965 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:41.261972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:41.262681 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:41.262685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:41.263668 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:41.263678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:41.264094 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:41.264097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:41.264182 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:41.264233 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:54:41.282874 (+ 18641us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 22:54:41.283096 (+ 222us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.114.254:41967: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":127,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.20.114.254:41967: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 22:54:41.285106 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 26069
2026-09-25T22:54:41Z chronyd exiting
I20260925 22:54:41.322502 20939 test_util.cc:175] -----------------------------------------------
I20260925 22:54:41.322590 20939 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376856092611-20939-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (309 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 22:54:41.323578 20939 mini_oidc.cc:152] Starting JWKS server
I20260925 22:54:41.324378 20939 webserver.cc:460] Webserver started at https://127.0.0.1:43023/ using document root <none> and password file <none>
I20260925 22:54:41.324455 20939 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 22:54:41.324671 20939 webserver.cc:460] Webserver started at http://127.0.0.1:46575/ using document root <none> and password file <none>
2026-09-25T22:54:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:54:41Z Disabled control of system clock
I20260925 22:54:41.333323 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:44453
--webserver_interface=127.20.114.254
--webserver_port=0
--builtin_ntp_servers=127.20.114.212:42329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:44453
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43023/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376856092611-20939-0/testchainca.cert with env {}
W20260925 22:54:41.446971 26172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:41.447273 26172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:41.447315 26172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:41.451304 26172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:41.451364 26172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:41.451380 26172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:41.451397 26172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:41.451412 26172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:54:41.451428 26172 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:54:41.451449 26172 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43023/jwks.json
I20260925 22:54:41.456797 26172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:42329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:44453
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:44453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43023/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376856092611-20939-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:41.457872 26172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:41.458837 26172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:41.467362 26180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:41.467376 26177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:41.467396 26178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:41.467634 26172 server_base.cc:1034] running on GCE node
I20260925 22:54:41.468204 26172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:41.469471 26172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:41.470640 26172 hybrid_clock.cc:648] HybridClock initialized: now 1790376881470606 us; error 50 us; skew 500 ppm
I20260925 22:54:41.472544 26172 webserver.cc:460] Webserver started at http://127.20.114.254:43911/ using document root <none> and password file <none>
I20260925 22:54:41.473204 26172 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:41.473269 26172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:41.473524 26172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:41.475647 26172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "0355ffe884e04f0688006b433ae2c73a"
format_stamp: "Formatted at 2026-09-25 22:54:41 on dist-test-slave-x1lh"
server_key: "721d1fdc4466636def28f95a5b92a929"
server_key_iv: "0b1802692047bd843bb7a8053a5081cc"
server_key_version: "encryptionkey@0"
I20260925 22:54:41.476233 26172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "0355ffe884e04f0688006b433ae2c73a"
format_stamp: "Formatted at 2026-09-25 22:54:41 on dist-test-slave-x1lh"
server_key: "721d1fdc4466636def28f95a5b92a929"
server_key_iv: "0b1802692047bd843bb7a8053a5081cc"
server_key_version: "encryptionkey@0"
I20260925 22:54:41.480036 26172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260925 22:54:41.482579 26186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:41.483865 26172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:54:41.484016 26172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "0355ffe884e04f0688006b433ae2c73a"
format_stamp: "Formatted at 2026-09-25 22:54:41 on dist-test-slave-x1lh"
server_key: "721d1fdc4466636def28f95a5b92a929"
server_key_iv: "0b1802692047bd843bb7a8053a5081cc"
server_key_version: "encryptionkey@0"
I20260925 22:54:41.484141 26172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:41.527654 26172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:41.528391 26172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:41.528599 26172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:41.535385 26172 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:44453
I20260925 22:54:41.535460 26238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:44453 every 8 connection(s)
I20260925 22:54:41.536374 26172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:54:41.539541 26239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:41.539803 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 26172
I20260925 22:54:41.539922 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:54:41.540203 20939 external_mini_cluster.cc:1428] Setting key 583735f66e4c4947c502d37071b88303
I20260925 22:54:41.546025 26239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0355ffe884e04f0688006b433ae2c73a: Bootstrap starting.
I20260925 22:54:41.548375 26239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0355ffe884e04f0688006b433ae2c73a: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:41.549191 26239 log.cc:824] T 00000000000000000000000000000000 P 0355ffe884e04f0688006b433ae2c73a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:41.549420 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:41.541552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.20.114.254:44453)
0925 22:54:41.542205 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:41.542219 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:41.542255 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:54:41.542941 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:41.542958 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:41.543350 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:41.543683 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:41.544097 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:41.545122 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:41.546036 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:41.546548 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:41.546724 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:41.547616 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:41.547621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:41.547634 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:54:41.548004 (+ 370us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:41.548066 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:54:41.548085 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:41.548190 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:41.548332 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 22:54:41.548602 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":145,"threads_started":1}
I20260925 22:54:41.551191 26239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0355ffe884e04f0688006b433ae2c73a: No bootstrap required, opened a new log
I20260925 22:54:41.552198 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:54:41.554952 26239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0355ffe884e04f0688006b433ae2c73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0355ffe884e04f0688006b433ae2c73a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44453 } }
I20260925 22:54:41.555172 26239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0355ffe884e04f0688006b433ae2c73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:41.555254 26239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0355ffe884e04f0688006b433ae2c73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0355ffe884e04f0688006b433ae2c73a, State: Initialized, Role: FOLLOWER
I20260925 22:54:41.555707 26239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0355ffe884e04f0688006b433ae2c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0355ffe884e04f0688006b433ae2c73a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44453 } }
I20260925 22:54:41.555845 26239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0355ffe884e04f0688006b433ae2c73a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:54:41.555923 26239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0355ffe884e04f0688006b433ae2c73a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:54:41.556048 26239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0355ffe884e04f0688006b433ae2c73a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:41.556986 26239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0355ffe884e04f0688006b433ae2c73a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0355ffe884e04f0688006b433ae2c73a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44453 } }
I20260925 22:54:41.557319 26239 leader_election.cc:302] T 00000000000000000000000000000000 P 0355ffe884e04f0688006b433ae2c73a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0355ffe884e04f0688006b433ae2c73a; no voters:
I20260925 22:54:41.557607 26239 leader_election.cc:288] T 00000000000000000000000000000000 P 0355ffe884e04f0688006b433ae2c73a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:54:41.557731 26244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0355ffe884e04f0688006b433ae2c73a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:41.557994 26244 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0355ffe884e04f0688006b433ae2c73a [term 1 LEADER]: Becoming Leader. State: Replica: 0355ffe884e04f0688006b433ae2c73a, State: Running, Role: LEADER
I20260925 22:54:41.558331 26244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0355ffe884e04f0688006b433ae2c73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0355ffe884e04f0688006b433ae2c73a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44453 } }
I20260925 22:54:41.558799 26239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0355ffe884e04f0688006b433ae2c73a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:41.559883 26245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0355ffe884e04f0688006b433ae2c73a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0355ffe884e04f0688006b433ae2c73a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0355ffe884e04f0688006b433ae2c73a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44453 } } }
I20260925 22:54:41.560014 26245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0355ffe884e04f0688006b433ae2c73a [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:41.560355 26246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0355ffe884e04f0688006b433ae2c73a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0355ffe884e04f0688006b433ae2c73a. Latest consensus state: current_term: 1 leader_uuid: "0355ffe884e04f0688006b433ae2c73a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0355ffe884e04f0688006b433ae2c73a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44453 } } }
I20260925 22:54:41.560441 26246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0355ffe884e04f0688006b433ae2c73a [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:41.560524 26253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:54:41.563239 26253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:54:41.568553 26253 catalog_manager.cc:1347] Generated new cluster ID: f2f10489cd584ca980bc182c0862abf3
I20260925 22:54:41.568617 26253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:41.582855 26253 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:54:41.583758 26253 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:41.588200 26253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0355ffe884e04f0688006b433ae2c73a: Generated new TSK 0
I20260925 22:54:41.588984 26253 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 22:54:41.619616 26269 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 22:54:41.619709 26269 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 22:54:41.620357 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:41.596959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53538 (local address 127.20.114.254:44453)
0925 22:54:41.597098 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:41.597102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:41.597297 (+ 195us) server_negotiation.cc:407] Connection header received
0925 22:54:41.597573 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:41.597577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:41.597635 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:41.597712 (+ 77us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:54:41.598055 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:41.598580 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:41.599842 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:41.600062 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:41.600134 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:41.600460 (+ 326us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:54:41.619983 (+ 19523us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43023/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 22:54:41.620231 (+ 248us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53538: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43023/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 22:54:41.620443 26242 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53538: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43023/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 22:54:41.620558 26268 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:41.596906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:44453 (local address 127.0.0.1:53538)
0925 22:54:41.597201 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:41.597217 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:41.597334 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:41.597839 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:41.597842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:41.597850 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:54:41.597953 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:41.597959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:41.598714 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:41.598718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:41.599719 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:41.599729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:41.600176 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:41.600180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:41.600268 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:41.600295 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:54:41.620197 (+ 19902us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43023/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 22:54:41.620387 (+ 190us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.114.254:44453: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43023/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":105,"threads_started":1}
I20260925 22:54:41.621349 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 26172
2026-09-25T22:54:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (336 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 22:54:41.660025 20939 mini_oidc.cc:152] Starting JWKS server
I20260925 22:54:41.660967 20939 webserver.cc:460] Webserver started at https://127.0.0.1:34551/ using document root <none> and password file <none>
I20260925 22:54:41.661091 20939 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 22:54:41.661363 20939 webserver.cc:460] Webserver started at http://127.0.0.1:38827/ using document root <none> and password file <none>
2026-09-25T22:54:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:54:41Z Disabled control of system clock
I20260925 22:54:41.670239 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:38183
--webserver_interface=127.20.114.254
--webserver_port=0
--builtin_ntp_servers=127.20.114.212:44567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:38183
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34551/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 22:54:41.780844 26275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:41.781191 26275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:41.781231 26275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:41.785120 26275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:41.785166 26275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:41.785181 26275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:41.785197 26275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:41.785212 26275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:54:41.785241 26275 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:54:41.785255 26275 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34551/jwks.json
I20260925 22:54:41.790071 26275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:44567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:38183
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:38183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34551/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:41.791051 26275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:41.792366 26275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:41.800560 26281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:41.800534 26283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:41.800587 26280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:41.800845 26275 server_base.cc:1034] running on GCE node
I20260925 22:54:41.801433 26275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:41.802418 26275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:41.803575 26275 hybrid_clock.cc:648] HybridClock initialized: now 1790376881803555 us; error 31 us; skew 500 ppm
I20260925 22:54:41.805423 26275 webserver.cc:460] Webserver started at http://127.20.114.254:34193/ using document root <none> and password file <none>
I20260925 22:54:41.805979 26275 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:41.806037 26275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:41.806298 26275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:41.808368 26275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "9297405c411f498aba6773fa9318ebde"
format_stamp: "Formatted at 2026-09-25 22:54:41 on dist-test-slave-x1lh"
server_key: "cb3fa7c869287dde354e63381f644145"
server_key_iv: "f5655e930a7493cba3c37c8af20276f5"
server_key_version: "encryptionkey@0"
I20260925 22:54:41.808986 26275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "9297405c411f498aba6773fa9318ebde"
format_stamp: "Formatted at 2026-09-25 22:54:41 on dist-test-slave-x1lh"
server_key: "cb3fa7c869287dde354e63381f644145"
server_key_iv: "f5655e930a7493cba3c37c8af20276f5"
server_key_version: "encryptionkey@0"
I20260925 22:54:41.812765 26275 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:54:41.815280 26289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:41.816460 26275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:41.816607 26275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "9297405c411f498aba6773fa9318ebde"
format_stamp: "Formatted at 2026-09-25 22:54:41 on dist-test-slave-x1lh"
server_key: "cb3fa7c869287dde354e63381f644145"
server_key_iv: "f5655e930a7493cba3c37c8af20276f5"
server_key_version: "encryptionkey@0"
I20260925 22:54:41.816718 26275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:41.832769 26275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:41.833436 26275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:41.833650 26275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:41.839968 26275 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:38183
I20260925 22:54:41.839980 26341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:38183 every 8 connection(s)
I20260925 22:54:41.840909 26275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:54:41.843963 26342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:41.845729 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 26275
I20260925 22:54:41.845860 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:54:41.846127 20939 external_mini_cluster.cc:1428] Setting key e1158de2430257f41f644912354e6b6f
I20260925 22:54:41.850358 26342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde: Bootstrap starting.
I20260925 22:54:41.852500 26342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:41.853274 26342 log.cc:824] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:41.855067 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:41.847511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48180 (local address 127.20.114.254:38183)
0925 22:54:41.848133 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:41.848148 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:41.848187 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:54:41.848816 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:41.848832 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:41.849278 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:41.849626 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:41.850032 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:41.851014 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:41.851891 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:41.852355 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:41.852520 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:41.853420 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:41.853426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:41.853442 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:54:41.853830 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:41.853886 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:54:41.853906 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:41.854012 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:41.854143 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 22:54:41.854389 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":127,"threads_started":1}
I20260925 22:54:41.855199 26342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde: No bootstrap required, opened a new log
I20260925 22:54:41.857836 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:54:41.859419 26342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9297405c411f498aba6773fa9318ebde" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38183 } }
I20260925 22:54:41.859702 26342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:41.859763 26342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9297405c411f498aba6773fa9318ebde, State: Initialized, Role: FOLLOWER
I20260925 22:54:41.860168 26342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9297405c411f498aba6773fa9318ebde" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38183 } }
I20260925 22:54:41.860316 26342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:54:41.860383 26342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:54:41.860481 26342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:41.861665 26342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9297405c411f498aba6773fa9318ebde" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38183 } }
I20260925 22:54:41.862001 26342 leader_election.cc:302] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9297405c411f498aba6773fa9318ebde; no voters:
I20260925 22:54:41.862310 26342 leader_election.cc:288] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:54:41.862448 26347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:41.862669 26347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde [term 1 LEADER]: Becoming Leader. State: Replica: 9297405c411f498aba6773fa9318ebde, State: Running, Role: LEADER
I20260925 22:54:41.862952 26347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9297405c411f498aba6773fa9318ebde" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38183 } }
I20260925 22:54:41.863585 26342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:41.864578 26349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9297405c411f498aba6773fa9318ebde. Latest consensus state: current_term: 1 leader_uuid: "9297405c411f498aba6773fa9318ebde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9297405c411f498aba6773fa9318ebde" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38183 } } }
I20260925 22:54:41.864717 26349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:41.864965 26348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9297405c411f498aba6773fa9318ebde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9297405c411f498aba6773fa9318ebde" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38183 } } }
I20260925 22:54:41.865073 26348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde [sys.catalog]: This master's current role is: LEADER
W20260925 22:54:41.868357 26363 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:54:41.868418 26363 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:54:41.868495 26356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:54:41.869266 26356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:54:41.874812 26356 catalog_manager.cc:1347] Generated new cluster ID: 669ba693512441cb91d297b22c455712
I20260925 22:54:41.874888 26356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:41.889580 26356 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:54:41.890861 26356 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:41.899724 26356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9297405c411f498aba6773fa9318ebde: Generated new TSK 0
I20260925 22:54:41.900549 26356 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:54:41.924701 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:41.918806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48182 (local address 127.20.114.254:38183)
0925 22:54:41.918934 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:41.918940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:41.919095 (+ 155us) server_negotiation.cc:407] Connection header received
0925 22:54:41.919244 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:41.919247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:41.919300 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:41.919417 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:54:41.919596 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:41.920459 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:41.921454 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:41.921803 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:41.921899 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:41.922171 (+ 272us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:54:41.924339 (+ 2168us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34551/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 22:54:41.924566 (+ 227us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48182: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34551/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 22:54:41.924773 26345 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48182: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34551/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 22:54:41.926324 26371 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:41.918696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:38183 (local address 127.0.0.1:48182)
0925 22:54:41.919010 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:41.919022 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:41.919128 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:41.919413 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:41.919415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:41.919421 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:54:41.919510 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:41.919514 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:41.920560 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:41.920563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:41.921352 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:41.921361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:41.921929 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:41.921931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:41.921990 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:41.922011 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:54:41.924512 (+ 2501us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34551/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 22:54:41.926163 (+ 1651us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.114.254:38183: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34551/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":1460,"thread_start_us":103,"threads_started":1}
I20260925 22:54:41.926654 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 26275
E20260925 22:54:41.935271 26372 webserver.cc:533] Webserver: sslize failed:
E20260925 22:54:41.935338 26372 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T22:54:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (305 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T22:54:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:54:41Z Disabled control of system clock
I20260925 22:54:41.973502 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:42451
--webserver_interface=127.20.114.254
--webserver_port=0
--builtin_ntp_servers=127.20.114.212:41817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:42451
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376856092611-20939-0/testchainca.cert with env {}
W20260925 22:54:42.081477 26376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:42.081750 26376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:42.081789 26376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:42.085821 26376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:42.085876 26376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:42.085892 26376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:42.085908 26376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:42.085923 26376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:54:42.085938 26376 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:54:42.085951 26376 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 22:54:42.091425 26376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:41817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:42451
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:42451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376856092611-20939-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:42.092418 26376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:42.093380 26376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:42.101572 26381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:42.101590 26382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:42.101595 26384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:42.101861 26376 server_base.cc:1034] running on GCE node
I20260925 22:54:42.102370 26376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:42.103466 26376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:42.104652 26376 hybrid_clock.cc:648] HybridClock initialized: now 1790376882104624 us; error 50 us; skew 500 ppm
I20260925 22:54:42.106419 26376 webserver.cc:460] Webserver started at http://127.20.114.254:44325/ using document root <none> and password file <none>
I20260925 22:54:42.106976 26376 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:42.107066 26376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:42.107288 26376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:42.109300 26376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "0896555348844f04a5e6476e1a440a54"
format_stamp: "Formatted at 2026-09-25 22:54:42 on dist-test-slave-x1lh"
server_key: "0885c757ac78d9d295ef31792b18a209"
server_key_iv: "bfd839980c7cef6ce5836d417b6129c5"
server_key_version: "encryptionkey@0"
I20260925 22:54:42.109865 26376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "0896555348844f04a5e6476e1a440a54"
format_stamp: "Formatted at 2026-09-25 22:54:42 on dist-test-slave-x1lh"
server_key: "0885c757ac78d9d295ef31792b18a209"
server_key_iv: "bfd839980c7cef6ce5836d417b6129c5"
server_key_version: "encryptionkey@0"
I20260925 22:54:42.113289 26376 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:54:42.115628 26390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:42.116772 26376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:42.116947 26376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "0896555348844f04a5e6476e1a440a54"
format_stamp: "Formatted at 2026-09-25 22:54:42 on dist-test-slave-x1lh"
server_key: "0885c757ac78d9d295ef31792b18a209"
server_key_iv: "bfd839980c7cef6ce5836d417b6129c5"
server_key_version: "encryptionkey@0"
I20260925 22:54:42.117069 26376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:42.134951 26376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:42.135684 26376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:42.135890 26376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:42.142753 26376 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:42451
I20260925 22:54:42.142765 26442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:42451 every 8 connection(s)
I20260925 22:54:42.143695 26376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:54:42.146505 26443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:42.148739 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 26376
I20260925 22:54:42.148883 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:54:42.149215 20939 external_mini_cluster.cc:1428] Setting key 22afed7d8652f3f8bfc51b5301328823
I20260925 22:54:42.152593 26443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0896555348844f04a5e6476e1a440a54: Bootstrap starting.
I20260925 22:54:42.154803 26443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0896555348844f04a5e6476e1a440a54: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:42.155524 26443 log.cc:824] T 00000000000000000000000000000000 P 0896555348844f04a5e6476e1a440a54: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:42.157533 26443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0896555348844f04a5e6476e1a440a54: No bootstrap required, opened a new log
I20260925 22:54:42.158232 26446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:42.150553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44436 (local address 127.20.114.254:42451)
0925 22:54:42.151159 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:42.151174 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:42.151206 (+ 32us) server_negotiation.cc:407] Connection header received
0925 22:54:42.151825 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:42.151841 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:42.152227 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:42.152560 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:42.153060 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:42.154000 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:42.154889 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:42.155400 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:42.155574 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:42.156469 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:42.156474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:42.156487 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:54:42.156901 (+ 414us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:42.156970 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:54:42.156991 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:42.157115 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:42.157228 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 22:54:42.157482 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":121,"threads_started":1}
I20260925 22:54:42.161510 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:54:42.162235 26443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0896555348844f04a5e6476e1a440a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0896555348844f04a5e6476e1a440a54" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 42451 } }
I20260925 22:54:42.162559 26443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0896555348844f04a5e6476e1a440a54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:42.162626 26443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0896555348844f04a5e6476e1a440a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0896555348844f04a5e6476e1a440a54, State: Initialized, Role: FOLLOWER
I20260925 22:54:42.163044 26443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0896555348844f04a5e6476e1a440a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0896555348844f04a5e6476e1a440a54" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 42451 } }
I20260925 22:54:42.163194 26443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0896555348844f04a5e6476e1a440a54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:54:42.163264 26443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0896555348844f04a5e6476e1a440a54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:54:42.163369 26443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0896555348844f04a5e6476e1a440a54 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:42.164196 26443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0896555348844f04a5e6476e1a440a54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0896555348844f04a5e6476e1a440a54" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 42451 } }
I20260925 22:54:42.164467 26443 leader_election.cc:302] T 00000000000000000000000000000000 P 0896555348844f04a5e6476e1a440a54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0896555348844f04a5e6476e1a440a54; no voters:
I20260925 22:54:42.164719 26443 leader_election.cc:288] T 00000000000000000000000000000000 P 0896555348844f04a5e6476e1a440a54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:54:42.164829 26448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0896555348844f04a5e6476e1a440a54 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:42.165112 26448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0896555348844f04a5e6476e1a440a54 [term 1 LEADER]: Becoming Leader. State: Replica: 0896555348844f04a5e6476e1a440a54, State: Running, Role: LEADER
I20260925 22:54:42.165460 26448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0896555348844f04a5e6476e1a440a54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0896555348844f04a5e6476e1a440a54" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 42451 } }
I20260925 22:54:42.165923 26443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0896555348844f04a5e6476e1a440a54 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:42.166998 26449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0896555348844f04a5e6476e1a440a54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0896555348844f04a5e6476e1a440a54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0896555348844f04a5e6476e1a440a54" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 42451 } } }
I20260925 22:54:42.167132 26449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0896555348844f04a5e6476e1a440a54 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:42.167515 26457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:54:42.167439 26450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0896555348844f04a5e6476e1a440a54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0896555348844f04a5e6476e1a440a54. Latest consensus state: current_term: 1 leader_uuid: "0896555348844f04a5e6476e1a440a54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0896555348844f04a5e6476e1a440a54" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 42451 } } }
I20260925 22:54:42.168268 26450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0896555348844f04a5e6476e1a440a54 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:42.170408 26457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:54:42.175916 26457 catalog_manager.cc:1347] Generated new cluster ID: b28d2bb213ad4feda6722d5fff1a4cd7
I20260925 22:54:42.175979 26457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:42.195075 26457 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:54:42.196310 26457 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:42.206462 26457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0896555348844f04a5e6476e1a440a54: Generated new TSK 0
I20260925 22:54:42.207278 26457 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:54:42.284353 26446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:42.219777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44450 (local address 127.20.114.254:42451)
0925 22:54:42.219908 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:42.219912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:42.220114 (+ 202us) server_negotiation.cc:407] Connection header received
0925 22:54:42.220264 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:42.220266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:42.220311 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:42.220449 (+ 138us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:54:42.220690 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:42.221292 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:42.222178 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:42.222405 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:42.222512 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:42.222820 (+ 308us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:54:42.283860 (+ 61040us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 22:54:42.284193 (+ 333us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44450: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 22:54:42.284452 26446 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44450: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 22:54:42.284489 26472 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:42.219729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:42451 (local address 127.0.0.1:44450)
0925 22:54:42.220024 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:42.220039 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:42.220151 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:42.220431 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:42.220434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:42.220443 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:54:42.220572 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:42.220580 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:42.221401 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:42.221405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:42.222061 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:42.222069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:42.222533 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:42.222537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:42.222637 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:42.222658 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:54:42.284119 (+ 61461us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 22:54:42.284302 (+ 183us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.114.254:42451: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":120,"threads_started":1}
I20260925 22:54:42.285235 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 26376
2026-09-25T22:54:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (338 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (123 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (122 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:54:42 dist-test-slave-x1lh krb5kdc[26484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:54:42 dist-test-slave-x1lh krb5kdc[26484](info): set up 2 sockets
Sep 25 22:54:42 dist-test-slave-x1lh krb5kdc[26484](info): commencing operation
krb5kdc: starting...
W20260925 22:54:43.581503 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:54:43 dist-test-slave-x1lh krb5kdc[26484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:54:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:54:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:54:43.712360 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:40127
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:40127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:43.822106 26497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:43.822422 26497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:43.822515 26497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:43.826534 26497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:43.826624 26497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:43.826661 26497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:43.826694 26497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:43.826736 26497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:43.832655 26497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:40127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:40127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:43.833890 26497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:43.835017 26497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:43.843773 26505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:43.843936 26497 server_base.cc:1034] running on GCE node
W20260925 22:54:43.843791 26502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:43.843801 26503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:43.844581 26497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:43.845685 26497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:43.846839 26497 hybrid_clock.cc:648] HybridClock initialized: now 1790376883846826 us; error 24 us; skew 500 ppm
Sep 25 22:54:43 dist-test-slave-x1lh krb5kdc[26484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376883, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:43.849570 26497 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:54:43.850378 26497 webserver.cc:460] Webserver started at http://127.20.114.254:39621/ using document root <none> and password file <none>
I20260925 22:54:43.850947 26497 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:43.851008 26497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:43.851186 26497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:43.853261 26497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "4fe849497692410699b805b711c1c931"
format_stamp: "Formatted at 2026-09-25 22:54:43 on dist-test-slave-x1lh"
server_key: "d60902c3fb6457e1c1332c9e687efd1b"
server_key_iv: "3999c77ccdd7fa91fe173300d6beda23"
server_key_version: "encryptionkey@0"
I20260925 22:54:43.853772 26497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "4fe849497692410699b805b711c1c931"
format_stamp: "Formatted at 2026-09-25 22:54:43 on dist-test-slave-x1lh"
server_key: "d60902c3fb6457e1c1332c9e687efd1b"
server_key_iv: "3999c77ccdd7fa91fe173300d6beda23"
server_key_version: "encryptionkey@0"
I20260925 22:54:43.857412 26497 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:54:43.859769 26512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:43.861109 26497 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:43.861261 26497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "4fe849497692410699b805b711c1c931"
format_stamp: "Formatted at 2026-09-25 22:54:43 on dist-test-slave-x1lh"
server_key: "d60902c3fb6457e1c1332c9e687efd1b"
server_key_iv: "3999c77ccdd7fa91fe173300d6beda23"
server_key_version: "encryptionkey@0"
I20260925 22:54:43.861371 26497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:43.874774 26497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:43.877803 26497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:43.878013 26497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:43.884995 26497 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:40127
I20260925 22:54:43.885030 26564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:40127 every 8 connection(s)
I20260925 22:54:43.885953 26497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:54:43.888684 26565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:43.888827 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 26497
I20260925 22:54:43.888971 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:54:43.889261 20939 external_mini_cluster.cc:1428] Setting key fc2328e9d14e7dcbeb1906b44254d731
I20260925 22:54:43.894995 26565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fe849497692410699b805b711c1c931: Bootstrap starting.
I20260925 22:54:43.897348 26565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fe849497692410699b805b711c1c931: Neither blocks nor log segments found. Creating new log.
Sep 25 22:54:43 dist-test-slave-x1lh krb5kdc[26484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:43.898252 26565 log.cc:824] T 00000000000000000000000000000000 P 4fe849497692410699b805b711c1c931: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:43.900403 26565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fe849497692410699b805b711c1c931: No bootstrap required, opened a new log
I20260925 22:54:43.903430 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:43.890693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39900 (local address 127.20.114.254:40127)
0925 22:54:43.891117 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:43.891136 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:43.891169 (+ 33us) server_negotiation.cc:407] Connection header received
0925 22:54:43.891926 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:43.891944 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:43.892266 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:43.892601 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:43.893424 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:43.894403 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:43.895322 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:43.895800 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:43.895970 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:43.898764 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:43.898770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:43.898786 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:43.898816 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:43.900516 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:43.900990 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:43.900996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:43.901007 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:43.901117 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:43.901413 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:43.901416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:43.901418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:43.901805 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:43.902005 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:43.902278 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 22:54:43.902558 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":125,"threads_started":1}
I20260925 22:54:43.904224 26565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fe849497692410699b805b711c1c931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe849497692410699b805b711c1c931" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40127 } }
I20260925 22:54:43.904477 26565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fe849497692410699b805b711c1c931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:43.904537 26565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fe849497692410699b805b711c1c931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe849497692410699b805b711c1c931, State: Initialized, Role: FOLLOWER
I20260925 22:54:43.905040 26565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fe849497692410699b805b711c1c931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe849497692410699b805b711c1c931" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40127 } }
I20260925 22:54:43.905201 26565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fe849497692410699b805b711c1c931 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:54:43.905260 26565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fe849497692410699b805b711c1c931 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:54:43.905350 26565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fe849497692410699b805b711c1c931 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:43.906374 26565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fe849497692410699b805b711c1c931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe849497692410699b805b711c1c931" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40127 } }
I20260925 22:54:43.906668 26565 leader_election.cc:302] T 00000000000000000000000000000000 P 4fe849497692410699b805b711c1c931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fe849497692410699b805b711c1c931; no voters:
I20260925 22:54:43.906932 26565 leader_election.cc:288] T 00000000000000000000000000000000 P 4fe849497692410699b805b711c1c931 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:54:43.907063 26570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fe849497692410699b805b711c1c931 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:43.907370 26570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fe849497692410699b805b711c1c931 [term 1 LEADER]: Becoming Leader. State: Replica: 4fe849497692410699b805b711c1c931, State: Running, Role: LEADER
I20260925 22:54:43.907706 26570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fe849497692410699b805b711c1c931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe849497692410699b805b711c1c931" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40127 } }
I20260925 22:54:43.908206 26565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fe849497692410699b805b711c1c931 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:43.909433 26572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe849497692410699b805b711c1c931 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fe849497692410699b805b711c1c931. Latest consensus state: current_term: 1 leader_uuid: "4fe849497692410699b805b711c1c931" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe849497692410699b805b711c1c931" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40127 } } }
I20260925 22:54:43.909438 26571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe849497692410699b805b711c1c931 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fe849497692410699b805b711c1c931" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe849497692410699b805b711c1c931" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40127 } } }
I20260925 22:54:43.909621 26572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe849497692410699b805b711c1c931 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:43.909643 26571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe849497692410699b805b711c1c931 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:43.910983 26579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:54:43.914067 26579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:54:43.921137 26579 catalog_manager.cc:1347] Generated new cluster ID: 6ae82eb3e93b4264b3ab9e13870963ad
I20260925 22:54:43.921239 26579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:43.929329 26579 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:54:43.930703 26579 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:43.935807 26579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fe849497692410699b805b711c1c931: Generated new TSK 0
I20260925 22:54:43.936676 26579 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.114.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
I20260925 22:54:43.964959 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:40127
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:44.075914 26591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:44.076186 26591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:44.076234 26591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:44.080394 26591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:44.080456 26591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:44.080514 26591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:54:44.086236 26591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:40127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:44.087304 26591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:44.088330 26591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:44.097815 26599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:44.097870 26596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:44.097849 26591 server_base.cc:1034] running on GCE node
W20260925 22:54:44.097815 26597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:44.098456 26591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:44.099097 26591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:44.100347 26591 hybrid_clock.cc:648] HybridClock initialized: now 1790376884100305 us; error 45 us; skew 500 ppm
Sep 25 22:54:44 dist-test-slave-x1lh krb5kdc[26484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376884, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:44.103269 26591 init.cc:377] Logged in from keytab as kudu/127.20.114.193@KRBTEST.COM (short username kudu)
I20260925 22:54:44.104099 26591 webserver.cc:460] Webserver started at http://127.20.114.193:39565/ using document root <none> and password file <none>
I20260925 22:54:44.104655 26591 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:44.104743 26591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:44.105017 26591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:44.107021 26591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "1d682588fabe46f5864c2a87deb22cc7"
format_stamp: "Formatted at 2026-09-25 22:54:44 on dist-test-slave-x1lh"
server_key: "ba11d33474812a0f850952fa6ad737de"
server_key_iv: "fc34138f807e5ee05453e783a4c27a28"
server_key_version: "encryptionkey@0"
I20260925 22:54:44.107574 26591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "1d682588fabe46f5864c2a87deb22cc7"
format_stamp: "Formatted at 2026-09-25 22:54:44 on dist-test-slave-x1lh"
server_key: "ba11d33474812a0f850952fa6ad737de"
server_key_iv: "fc34138f807e5ee05453e783a4c27a28"
server_key_version: "encryptionkey@0"
I20260925 22:54:44.111369 26591 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:54:44.113721 26606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:44.114892 26591 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260925 22:54:44.115036 26591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "1d682588fabe46f5864c2a87deb22cc7"
format_stamp: "Formatted at 2026-09-25 22:54:44 on dist-test-slave-x1lh"
server_key: "ba11d33474812a0f850952fa6ad737de"
server_key_iv: "fc34138f807e5ee05453e783a4c27a28"
server_key_version: "encryptionkey@0"
I20260925 22:54:44.115149 26591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:44.129734 26591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:44.132807 26591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:44.133034 26591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:44.133620 26591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:44.135021 26591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:44.135213 26591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:44.135280 26591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:44.135304 26591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:44.143994 26591 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:34503
I20260925 22:54:44.144011 26719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:34503 every 8 connection(s)
I20260925 22:54:44.144948 26591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:54:44.151343 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 26591
I20260925 22:54:44.151448 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:54:44.151723 20939 external_mini_cluster.cc:1428] Setting key 903bf91e5eab0025af2378d040fd1df4
Sep 25 22:54:44 dist-test-slave-x1lh krb5kdc[26484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376884, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:44.159237 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:44.146765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:43529 (local address 127.20.114.254:40127)
0925 22:54:44.146918 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:44.146923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:44.147792 (+ 869us) server_negotiation.cc:407] Connection header received
0925 22:54:44.149118 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:44.149121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:44.149169 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:44.149243 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:44.150382 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:44.150909 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:44.152285 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:44.152567 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:44.152722 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:44.155256 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:44.155261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:44.155266 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:44.155299 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:44.157055 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:44.157533 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:44.157539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:44.157542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:44.157642 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:44.157953 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:44.157958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:44.157962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:44.158228 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:44.158375 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:44.158827 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 22:54:44.158987 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:54:44.159814 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:44.147196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:40127 (local address 127.20.114.193:43529)
0925 22:54:44.147634 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:44.147676 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:44.148855 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:44.149398 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:44.149414 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:44.149757 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:44.150254 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:44.150267 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:44.151017 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:44.151020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:44.152176 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:44.152187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:44.152757 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:44.152761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:44.153007 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:44.153671 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:44.153687 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:44.155077 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:44.157176 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:44.157180 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:44.157192 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:44.157435 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:44.157737 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:44.157739 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:44.157740 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:44.157862 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:44.158364 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:44.158369 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:44.158585 (+ 216us) client_negotiation.cc:769] Sending connection context
0925 22:54:44.158739 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 22:54:44.158973 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":110,"threads_started":1}
I20260925 22:54:44.161041 26720 heartbeater.cc:347] Connected to a master server at 127.20.114.254:40127
I20260925 22:54:44.161285 26720 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.20.114.194@KRBTEST.COM; defaulting to no policy
I20260925 22:54:44.161789 26720 heartbeater.cc:511] Master 127.20.114.254:40127 requested a full tablet report, sending...
I20260925 22:54:44.163273 26529 ts_manager.cc:194] Registered new tserver with Master: 1d682588fabe46f5864c2a87deb22cc7 (127.20.114.193:34503)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.194@KRBTEST.COM" created.
I20260925 22:54:44.164570 26529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:43529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
I20260925 22:54:44.179970 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:40127
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:44.288673 26725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:44.289002 26725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:44.289079 26725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:44.293099 26725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:44.293198 26725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:44.293290 26725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:54:44.298712 26725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:40127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:44.299769 26725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:44.300823 26725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:44.310700 26730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:44.310686 26731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:44.310711 26733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:44.311021 26725 server_base.cc:1034] running on GCE node
I20260925 22:54:44.311410 26725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:44.312040 26725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:44.313200 26725 hybrid_clock.cc:648] HybridClock initialized: now 1790376884313166 us; error 44 us; skew 500 ppm
Sep 25 22:54:44 dist-test-slave-x1lh krb5kdc[26484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376884, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:44.315829 26725 init.cc:377] Logged in from keytab as kudu/127.20.114.194@KRBTEST.COM (short username kudu)
I20260925 22:54:44.316721 26725 webserver.cc:460] Webserver started at http://127.20.114.194:41563/ using document root <none> and password file <none>
I20260925 22:54:44.317337 26725 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:44.317426 26725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:44.317658 26725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:44.319809 26725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "9f6a77bdb7d948cd8b4c2a37e3259056"
format_stamp: "Formatted at 2026-09-25 22:54:44 on dist-test-slave-x1lh"
server_key: "36411afa3907f3698a69b9ad7c5c88b2"
server_key_iv: "404430e39b97ade9cc63d8697ea28b16"
server_key_version: "encryptionkey@0"
I20260925 22:54:44.320394 26725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "9f6a77bdb7d948cd8b4c2a37e3259056"
format_stamp: "Formatted at 2026-09-25 22:54:44 on dist-test-slave-x1lh"
server_key: "36411afa3907f3698a69b9ad7c5c88b2"
server_key_iv: "404430e39b97ade9cc63d8697ea28b16"
server_key_version: "encryptionkey@0"
I20260925 22:54:44.324136 26725 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 22:54:44.326596 26740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:44.327901 26725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:54:44.328040 26725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "9f6a77bdb7d948cd8b4c2a37e3259056"
format_stamp: "Formatted at 2026-09-25 22:54:44 on dist-test-slave-x1lh"
server_key: "36411afa3907f3698a69b9ad7c5c88b2"
server_key_iv: "404430e39b97ade9cc63d8697ea28b16"
server_key_version: "encryptionkey@0"
I20260925 22:54:44.328164 26725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:44.338750 26725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:44.342278 26725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:44.342510 26725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:44.343159 26725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:44.344036 26725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:44.344125 26725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:44.344204 26725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:44.344261 26725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:44.353518 26725 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:35511
I20260925 22:54:44.353536 26853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:35511 every 8 connection(s)
I20260925 22:54:44.354451 26725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:54:44.356334 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 26725
I20260925 22:54:44.356441 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:54:44.356714 20939 external_mini_cluster.cc:1428] Setting key 1c6b30d0132dd943a04393875676a298
Sep 25 22:54:44 dist-test-slave-x1lh krb5kdc[26484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376884, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.114.195@KRBTEST.COM; defaulting to no policy
I20260925 22:54:44.370304 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:44.356350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:46213 (local address 127.20.114.254:40127)
0925 22:54:44.356485 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:44.356488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:44.357571 (+ 1083us) server_negotiation.cc:407] Connection header received
0925 22:54:44.359333 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:44.359339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:44.359403 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:44.359461 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:44.361226 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:44.362104 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:44.363339 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:44.363774 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:44.363902 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:44.366863 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:44.366866 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:44.366868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:44.366891 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:44.368304 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:44.368811 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:44.368814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:44.368815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:44.368961 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:44.369288 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:44.369290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:44.369291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:44.369484 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:44.369560 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:44.370073 (+ 513us) server_negotiation.cc:299] Negotiation successful
0925 22:54:44.370180 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:54:44.371270 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:44.356921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:40127 (local address 127.20.114.194:46213)
0925 22:54:44.357422 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:44.357474 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:44.359037 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:44.359913 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:44.359925 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:44.360430 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:44.361067 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:44.361089 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:44.362242 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:44.362244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:44.363215 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:44.363224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:44.363941 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:44.363944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:44.364079 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:44.364761 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:44.364778 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:44.366667 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:44.368407 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:44.368411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:44.368424 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:44.368717 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:44.369048 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:44.369050 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:44.369052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:44.369191 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:44.369571 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:44.369576 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:44.369828 (+ 252us) client_negotiation.cc:769] Sending connection context
0925 22:54:44.370000 (+ 172us) client_negotiation.cc:240] Negotiation successful
0925 22:54:44.370266 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":139,"threads_started":1}
I20260925 22:54:44.372452 26854 heartbeater.cc:347] Connected to a master server at 127.20.114.254:40127
I20260925 22:54:44.372713 26854 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:44.373284 26854 heartbeater.cc:511] Master 127.20.114.254:40127 requested a full tablet report, sending...
I20260925 22:54:44.374403 26529 ts_manager.cc:194] Registered new tserver with Master: 9f6a77bdb7d948cd8b4c2a37e3259056 (127.20.114.194:35511)
I20260925 22:54:44.374966 26529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:46213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.195@KRBTEST.COM" created.
I20260925 22:54:45.166960 26720 heartbeater.cc:503] Master 127.20.114.254:40127 was elected leader, sending a full tablet report...
I20260925 22:54:45.377121 26854 heartbeater.cc:503] Master 127.20.114.254:40127 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
W20260925 22:54:45.578605 20939 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.20.114.195: real 1.220s user 0.001s sys 0.003s
I20260925 22:54:45.579142 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:40127
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:45.692996 26859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:45.693259 26859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:45.693302 26859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:45.697388 26859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:45.697439 26859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:45.697496 26859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:54:45.702591 26859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:40127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:45.703608 26859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:45.704656 26859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:45.714723 26865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:45.714735 26864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:45.714928 26859 server_base.cc:1034] running on GCE node
W20260925 22:54:45.714720 26867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:45.715420 26859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:45.716133 26859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:45.717315 26859 hybrid_clock.cc:648] HybridClock initialized: now 1790376885717298 us; error 33 us; skew 500 ppm
Sep 25 22:54:45 dist-test-slave-x1lh krb5kdc[26484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376885, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:45.723402 26859 init.cc:377] Logged in from keytab as kudu/127.20.114.195@KRBTEST.COM (short username kudu)
I20260925 22:54:45.724299 26859 webserver.cc:460] Webserver started at http://127.20.114.195:44579/ using document root <none> and password file <none>
I20260925 22:54:45.724908 26859 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:45.725004 26859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:45.725222 26859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:45.727361 26859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "2188cd1aafaf445aa08e8d860e8f4b38"
format_stamp: "Formatted at 2026-09-25 22:54:45 on dist-test-slave-x1lh"
server_key: "0eb239f0f23ab8c3810bbda09fbd8c4b"
server_key_iv: "85b43731cea9e0dbd10424fa8bee1568"
server_key_version: "encryptionkey@0"
I20260925 22:54:45.727928 26859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "2188cd1aafaf445aa08e8d860e8f4b38"
format_stamp: "Formatted at 2026-09-25 22:54:45 on dist-test-slave-x1lh"
server_key: "0eb239f0f23ab8c3810bbda09fbd8c4b"
server_key_iv: "85b43731cea9e0dbd10424fa8bee1568"
server_key_version: "encryptionkey@0"
I20260925 22:54:45.731809 26859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 22:54:45.734592 26874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:45.735812 26859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:54:45.735926 26859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "2188cd1aafaf445aa08e8d860e8f4b38"
format_stamp: "Formatted at 2026-09-25 22:54:45 on dist-test-slave-x1lh"
server_key: "0eb239f0f23ab8c3810bbda09fbd8c4b"
server_key_iv: "85b43731cea9e0dbd10424fa8bee1568"
server_key_version: "encryptionkey@0"
I20260925 22:54:45.736025 26859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:45.755141 26859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:45.758208 26859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:45.758401 26859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:45.759009 26859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:45.759862 26859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:45.759908 26859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:45.759974 26859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:45.760010 26859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:45.769237 26859 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:38041
I20260925 22:54:45.769254 26987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:38041 every 8 connection(s)
I20260925 22:54:45.770171 26859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:54:45.775844 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 26859
I20260925 22:54:45.775980 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:54:45.776248 20939 external_mini_cluster.cc:1428] Setting key 249813dad81092e9ab21978ab597a661
Sep 25 22:54:45 dist-test-slave-x1lh krb5kdc[26484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376885, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:45.784547 26990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:45.771960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:50109 (local address 127.20.114.254:40127)
0925 22:54:45.772270 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:45.772277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:45.773041 (+ 764us) server_negotiation.cc:407] Connection header received
0925 22:54:45.775026 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:45.775030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:45.775084 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:45.775180 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:45.776522 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:45.777147 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:45.778273 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:45.778503 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:45.778616 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:45.781024 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:45.781027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:45.781029 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:45.781052 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:45.782480 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:45.783007 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:45.783012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:45.783014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:45.783100 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:45.783451 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:45.783455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:45.783456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:45.783640 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:45.783747 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:45.784240 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 22:54:45.784366 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":87,"threads_started":1}
I20260925 22:54:45.785393 26991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:45.772325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:40127 (local address 127.20.114.195:50109)
0925 22:54:45.772840 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:45.772925 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:45.774769 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:45.775387 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:45.775394 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:45.775762 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:45.776319 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:45.776356 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:45.777290 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:45.777292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:45.778151 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:45.778159 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:45.778659 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:45.778663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:45.778825 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:45.779420 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:45.779436 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:45.780815 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:45.782621 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:45.782628 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:45.782643 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:45.782899 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:45.783194 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:45.783198 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:45.783200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:45.783344 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:45.783754 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:45.783763 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:45.784004 (+ 241us) client_negotiation.cc:769] Sending connection context
0925 22:54:45.784187 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 22:54:45.784507 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":105,"threads_started":1}
I20260925 22:54:45.786531 26988 heartbeater.cc:347] Connected to a master server at 127.20.114.254:40127
I20260925 22:54:45.786734 26988 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:45.787168 26988 heartbeater.cc:511] Master 127.20.114.254:40127 requested a full tablet report, sending...
I20260925 22:54:45.788115 26529 ts_manager.cc:194] Registered new tserver with Master: 2188cd1aafaf445aa08e8d860e8f4b38 (127.20.114.195:38041)
I20260925 22:54:45.788796 26529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:50109
I20260925 22:54:45.790175 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/test-admin.keytab.
W20260925 22:54:46.160643 20939 mini_kdc.cc:302] Time spent creating keytab for test-admin: real 0.370s user 0.001s sys 0.001s
Sep 25 22:54:46 dist-test-slave-x1lh krb5kdc[26484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:46.162748 20939 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 22:54:46.165639 26996 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:46.164118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48850 (local address 127.20.114.193:34503)
0925 22:54:46.164427 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:46.164445 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:46.164476 (+ 31us) server_negotiation.cc:407] Connection header received
0925 22:54:46.164525 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:46.164530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:46.164718 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:46.164906 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:46.165449 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48850: BlockingRecv error: recv got EOF from 127.0.0.1:48850 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":145,"threads_started":1}
W20260925 22:54:46.165663 20939 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/krb5cc' of type FILE
Sep 25 22:54:46 dist-test-slave-x1lh krb5kdc[26484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:46.166692 20939 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 22:54:46 dist-test-slave-x1lh krb5kdc[26484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:46.167595 20939 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 22:54:46 dist-test-slave-x1lh krb5kdc[26484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.193@KRBTEST.COM
I20260925 22:54:46.178550 26996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:46.168570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48858 (local address 127.20.114.193:34503)
0925 22:54:46.168700 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:46.168710 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:46.168782 (+ 72us) server_negotiation.cc:407] Connection header received
0925 22:54:46.169106 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:46.169109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:46.169154 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:46.169235 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:46.169891 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:46.170780 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:46.171587 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:46.171973 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:46.172065 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:46.174735 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:46.174745 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:46.174798 (+ 53us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:46.174823 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:46.176776 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:46.177329 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:46.177334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:46.177336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:46.177412 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:46.177653 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:46.177654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:46.177655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:46.177867 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:46.178012 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:46.178201 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 22:54:46.178357 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 22:54:46.182457 26996 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:46.181429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.20.114.193:34503)
0925 22:54:46.181539 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:46.181543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:46.181721 (+ 178us) server_negotiation.cc:407] Connection header received
0925 22:54:46.181849 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:46.181852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:46.181897 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:46.181976 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:46.182361 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48870: BlockingRecv error: recv got EOF from 127.0.0.1:48870 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260925 22:54:46.182802 20939 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/krb5cc' of type FILE
Sep 25 22:54:46 dist-test-slave-x1lh krb5kdc[26484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:46.183866 20939 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 22:54:46 dist-test-slave-x1lh krb5kdc[26484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:46.184773 20939 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 22:54:46 dist-test-slave-x1lh krb5kdc[26484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.193@KRBTEST.COM
I20260925 22:54:46.193679 26996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:46.185693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48874 (local address 127.20.114.193:34503)
0925 22:54:46.185812 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:46.185816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:46.185927 (+ 111us) server_negotiation.cc:407] Connection header received
0925 22:54:46.186077 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:46.186086 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:46.186121 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:46.186204 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:46.186834 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:46.187622 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:46.188354 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:46.188570 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:46.188643 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:46.190734 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:46.190737 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:46.190739 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:46.190760 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:46.192228 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:46.192575 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:46.192577 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:46.192579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:46.192646 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:46.192859 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:46.192922 (+ 63us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:46.192923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:46.193089 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:46.193195 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:46.193370 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 22:54:46.193455 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 22:54:46.196357 26996 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:46.195526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48880 (local address 127.20.114.193:34503)
0925 22:54:46.195652 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:46.195656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:46.195671 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:54:46.195711 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:46.195713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:46.195745 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:46.195828 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:46.196268 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48880: BlockingRecv error: recv got EOF from 127.0.0.1:48880 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 22:54:46.196570 20939 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376856092611-20939-0/krb5kdc/krb5cc' of type FILE
Sep 25 22:54:46 dist-test-slave-x1lh krb5kdc[26484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:46.197605 20939 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 22:54:46 dist-test-slave-x1lh krb5kdc[26484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:46.198575 20939 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 22:54:46 dist-test-slave-x1lh krb5kdc[26484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.193@KRBTEST.COM
I20260925 22:54:46.207690 26996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:46.199480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48890 (local address 127.20.114.193:34503)
0925 22:54:46.199631 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:46.199636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:46.199769 (+ 133us) server_negotiation.cc:407] Connection header received
0925 22:54:46.199934 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:46.199937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:46.199983 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:46.200073 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:46.200596 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:46.201235 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:46.202097 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:46.202336 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:46.202455 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:46.204687 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:46.204690 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:46.204692 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:46.204712 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:46.206214 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:46.206641 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:46.206644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:46.206645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:46.206715 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:46.207018 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:46.207020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:46.207021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:46.207172 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:46.207269 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:46.207498 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 22:54:46.207581 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 22:54:46.208698 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 26591
I20260925 22:54:46.214138 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 26725
I20260925 22:54:46.219429 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 26859
I20260925 22:54:46.224392 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 26497
2026-09-25T22:54:46Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3695 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:54:46 dist-test-slave-x1lh krb5kdc[27009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:54:46 dist-test-slave-x1lh krb5kdc[27009](info): set up 2 sockets
Sep 25 22:54:46 dist-test-slave-x1lh krb5kdc[27009](info): commencing operation
krb5kdc: starting...
W20260925 22:54:47.265305 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:54:47 dist-test-slave-x1lh krb5kdc[27009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:54:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:54:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.254.keytab.
Entry for principal oryx/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.254.keytab.
I20260925 22:54:47.389859 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:37819
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.254.keytab
--principal=oryx/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:37819
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:47.500627 27022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:47.500921 27022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:47.501000 27022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:47.505343 27022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:47.505404 27022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:47.505429 27022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:47.505478 27022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:47.505501 27022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:47.511202 27022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:37819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.254.keytab
--principal=oryx/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:37819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:47.512316 27022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:47.513370 27022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:47.521770 27028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:47.521849 27030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:47.521780 27027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:47.522192 27022 server_base.cc:1034] running on GCE node
I20260925 22:54:47.522784 27022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:47.523856 27022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:47.525025 27022 hybrid_clock.cc:648] HybridClock initialized: now 1790376887525012 us; error 27 us; skew 500 ppm
Sep 25 22:54:47 dist-test-slave-x1lh krb5kdc[27009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376887, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:47.527997 27022 init.cc:377] Logged in from keytab as oryx/127.20.114.254@KRBTEST.COM (short username oryx)
I20260925 22:54:47.529273 27022 webserver.cc:460] Webserver started at http://127.20.114.254:34077/ using document root <none> and password file <none>
I20260925 22:54:47.530079 27022 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:47.530175 27022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:47.530453 27022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:47.533368 27022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "c96778fffbfd40f186940fb51039e3ae"
format_stamp: "Formatted at 2026-09-25 22:54:47 on dist-test-slave-x1lh"
server_key: "642a6257e616cd630b2f86ae3dce804a"
server_key_iv: "bba9af7b082a3f17518bcdf5388fbabb"
server_key_version: "encryptionkey@0"
I20260925 22:54:47.533953 27022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "c96778fffbfd40f186940fb51039e3ae"
format_stamp: "Formatted at 2026-09-25 22:54:47 on dist-test-slave-x1lh"
server_key: "642a6257e616cd630b2f86ae3dce804a"
server_key_iv: "bba9af7b082a3f17518bcdf5388fbabb"
server_key_version: "encryptionkey@0"
I20260925 22:54:47.537842 27022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:54:47.540148 27037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:47.541407 27022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:47.541529 27022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "c96778fffbfd40f186940fb51039e3ae"
format_stamp: "Formatted at 2026-09-25 22:54:47 on dist-test-slave-x1lh"
server_key: "642a6257e616cd630b2f86ae3dce804a"
server_key_iv: "bba9af7b082a3f17518bcdf5388fbabb"
server_key_version: "encryptionkey@0"
I20260925 22:54:47.541642 27022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:47.566594 27022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:47.569803 27022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:47.570008 27022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:47.577275 27022 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:37819
I20260925 22:54:47.577585 27089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:37819 every 8 connection(s)
I20260925 22:54:47.578279 27022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:54:47.581001 27090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:47.586259 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 27022
I20260925 22:54:47.586393 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:54:47.586617 27090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae: Bootstrap starting.
I20260925 22:54:47.586702 20939 external_mini_cluster.cc:1428] Setting key 4e00487dcc3ce7492105ac8417e4aa60
I20260925 22:54:47.589172 27090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:47.589895 27090 log.cc:824] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:47.591720 27090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae: No bootstrap required, opened a new log
Sep 25 22:54:47 dist-test-slave-x1lh krb5kdc[27009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM
I20260925 22:54:47.596233 27090 raft_consensus.cc:348] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } }
I20260925 22:54:47.596479 27090 raft_consensus.cc:374] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:47.596557 27090 raft_consensus.cc:729] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c96778fffbfd40f186940fb51039e3ae, State: Initialized, Role: FOLLOWER
I20260925 22:54:47.597064 27090 consensus_queue.cc:260] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } }
I20260925 22:54:47.597223 27090 raft_consensus.cc:388] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:54:47.597309 27090 raft_consensus.cc:482] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:54:47.597410 27090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:47.598356 27090 raft_consensus.cc:504] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } }
I20260925 22:54:47.598731 27090 leader_election.cc:302] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c96778fffbfd40f186940fb51039e3ae; no voters:
I20260925 22:54:47.599102 27090 leader_election.cc:288] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:54:47.599193 27095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:47.599431 27095 raft_consensus.cc:686] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [term 1 LEADER]: Becoming Leader. State: Replica: c96778fffbfd40f186940fb51039e3ae, State: Running, Role: LEADER
I20260925 22:54:47.599742 27095 consensus_queue.cc:237] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } }
I20260925 22:54:47.600481 27090 sys_catalog.cc:564] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:47.601529 27097 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader c96778fffbfd40f186940fb51039e3ae. Latest consensus state: current_term: 1 leader_uuid: "c96778fffbfd40f186940fb51039e3ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } }
I20260925 22:54:47.601662 27097 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:47.601905 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:47.588107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59104 (local address 127.20.114.254:37819)
0925 22:54:47.588503 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:47.588529 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:47.588565 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:54:47.589320 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:47.589339 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:47.589699 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:47.590046 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:47.590818 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:47.591791 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:47.592974 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:47.593458 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:47.593576 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:47.596467 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:47.596472 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:47.596510 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:47.596549 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:47.598455 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:47.598996 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:47.599004 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:47.599036 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:47.599183 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:47.599521 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:47.599523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:47.599525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:47.600008 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:47.600208 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:47.600501 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 22:54:47.600819 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":140,"threads_started":1}
I20260925 22:54:47.601994 27096 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c96778fffbfd40f186940fb51039e3ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } }
I20260925 22:54:47.602110 27096 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:47.602408 27104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:54:47.605317 27104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:54:47.610190 27104 catalog_manager.cc:1347] Generated new cluster ID: 0caef3ffb432457ba13ec0443bf9ca8a
I20260925 22:54:47.610257 27104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:47.649035 27104 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:54:47.650188 27104 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:47.657797 27104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae: Generated new TSK 0
I20260925 22:54:47.658615 27104 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.114.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.114.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.114.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.193.keytab.
Entry for principal oryx/127.20.114.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.193.keytab.
I20260925 22:54:47.685755 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37819
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.193.keytab
--principal=oryx/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:47.803153 27116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:47.803396 27116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:47.803493 27116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:47.807555 27116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:47.807614 27116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:47.807704 27116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:54:47.813181 27116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.193.keytab
--principal=oryx/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:47.814266 27116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:47.815287 27116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:47.824829 27121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:47.824836 27122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:47.824846 27124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:47.825124 27116 server_base.cc:1034] running on GCE node
I20260925 22:54:47.825496 27116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:47.826134 27116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:47.827309 27116 hybrid_clock.cc:648] HybridClock initialized: now 1790376887827279 us; error 41 us; skew 500 ppm
Sep 25 22:54:47 dist-test-slave-x1lh krb5kdc[27009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376887, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:47.830139 27116 init.cc:377] Logged in from keytab as oryx/127.20.114.193@KRBTEST.COM (short username oryx)
I20260925 22:54:47.831053 27116 webserver.cc:460] Webserver started at http://127.20.114.193:40751/ using document root <none> and password file <none>
I20260925 22:54:47.831653 27116 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:47.831753 27116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:47.831992 27116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:47.834093 27116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "c01ee4ba045949eebcc2660e769f7b6f"
format_stamp: "Formatted at 2026-09-25 22:54:47 on dist-test-slave-x1lh"
server_key: "86cdc06759ab8f71d89b53d01f48d025"
server_key_iv: "5366ff965476c4bec1502a6116c858e6"
server_key_version: "encryptionkey@0"
I20260925 22:54:47.834663 27116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "c01ee4ba045949eebcc2660e769f7b6f"
format_stamp: "Formatted at 2026-09-25 22:54:47 on dist-test-slave-x1lh"
server_key: "86cdc06759ab8f71d89b53d01f48d025"
server_key_iv: "5366ff965476c4bec1502a6116c858e6"
server_key_version: "encryptionkey@0"
I20260925 22:54:47.838331 27116 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:54:47.840818 27131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:47.842234 27116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:54:47.842371 27116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "c01ee4ba045949eebcc2660e769f7b6f"
format_stamp: "Formatted at 2026-09-25 22:54:47 on dist-test-slave-x1lh"
server_key: "86cdc06759ab8f71d89b53d01f48d025"
server_key_iv: "5366ff965476c4bec1502a6116c858e6"
server_key_version: "encryptionkey@0"
I20260925 22:54:47.842486 27116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:47.861326 27116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:47.864759 27116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:47.864995 27116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:47.865619 27116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:47.866448 27116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:47.866523 27116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:47.866591 27116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:47.866636 27116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:47.875521 27116 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:33393
I20260925 22:54:47.875543 27244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:33393 every 8 connection(s)
I20260925 22:54:47.876437 27116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:54:47.882302 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 27116
I20260925 22:54:47.882418 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:54:47.882700 20939 external_mini_cluster.cc:1428] Setting key ace7ea4d7381a55bf2b179fa3562fa0f
Sep 25 22:54:47 dist-test-slave-x1lh krb5kdc[27009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376887, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM
I20260925 22:54:47.890763 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:47.878287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:53509 (local address 127.20.114.254:37819)
0925 22:54:47.878430 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:47.878434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:47.879213 (+ 779us) server_negotiation.cc:407] Connection header received
0925 22:54:47.880562 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:47.880565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:47.880624 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:47.880736 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:47.881915 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:47.882732 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:47.884103 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:47.884342 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:47.884457 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:47.886980 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:47.886983 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:47.886985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:47.887008 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:47.888508 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:47.889174 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:47.889178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:47.889180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:47.889262 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:47.889641 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:47.889644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:47.889646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:47.889826 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:47.889934 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:47.890459 (+ 525us) server_negotiation.cc:299] Negotiation successful
0925 22:54:47.890593 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:54:47.891551 27247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:47.878609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.20.114.193:53509)
0925 22:54:47.879057 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:47.879102 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:47.880348 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:47.880927 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:47.880933 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:47.881293 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:47.881761 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:47.881774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:47.882877 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:47.882880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:47.883966 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:47.883978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:47.884492 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:47.884496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:47.884674 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:47.885389 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:47.885405 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:47.886807 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:47.888663 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:47.888669 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:47.888686 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:47.889070 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:47.889365 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:47.889367 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:47.889369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:47.889530 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:47.889942 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:47.889950 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:47.890218 (+ 268us) client_negotiation.cc:769] Sending connection context
0925 22:54:47.890406 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 22:54:47.890737 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for oryx/127.20.114.194@KRBTEST.COM; defaulting to no policy
I20260925 22:54:47.892776 27245 heartbeater.cc:347] Connected to a master server at 127.20.114.254:37819
I20260925 22:54:47.893049 27245 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:47.893529 27245 heartbeater.cc:511] Master 127.20.114.254:37819 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.114.194@KRBTEST.COM" created.
I20260925 22:54:47.895350 27054 ts_manager.cc:194] Registered new tserver with Master: c01ee4ba045949eebcc2660e769f7b6f (127.20.114.193:33393)
I20260925 22:54:47.896551 27054 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:53509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.114.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.194.keytab.
Entry for principal oryx/127.20.114.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.194.keytab.
I20260925 22:54:47.910059 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37819
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.194.keytab
--principal=oryx/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:48.019704 27250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:48.019946 27250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:48.019991 27250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:48.023923 27250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:48.023980 27250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:48.024037 27250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:54:48.029179 27250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.194.keytab
--principal=oryx/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:48.030179 27250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:48.031160 27250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:48.040663 27258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:48.040663 27256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:48.040704 27255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:48.040890 27250 server_base.cc:1034] running on GCE node
I20260925 22:54:48.041342 27250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:48.041986 27250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:48.043167 27250 hybrid_clock.cc:648] HybridClock initialized: now 1790376888043156 us; error 30 us; skew 500 ppm
Sep 25 22:54:48 dist-test-slave-x1lh krb5kdc[27009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376888, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:48.046053 27250 init.cc:377] Logged in from keytab as oryx/127.20.114.194@KRBTEST.COM (short username oryx)
I20260925 22:54:48.047032 27250 webserver.cc:460] Webserver started at http://127.20.114.194:34629/ using document root <none> and password file <none>
I20260925 22:54:48.047600 27250 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:48.047657 27250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:48.047874 27250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:48.049991 27250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "92bfd5c259204d5994cfd0029b038c21"
format_stamp: "Formatted at 2026-09-25 22:54:48 on dist-test-slave-x1lh"
server_key: "8a000aa7209f7ea6ca511865a72c311d"
server_key_iv: "ab611e92164c515cf2b3677d94ee5463"
server_key_version: "encryptionkey@0"
I20260925 22:54:48.050562 27250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "92bfd5c259204d5994cfd0029b038c21"
format_stamp: "Formatted at 2026-09-25 22:54:48 on dist-test-slave-x1lh"
server_key: "8a000aa7209f7ea6ca511865a72c311d"
server_key_iv: "ab611e92164c515cf2b3677d94ee5463"
server_key_version: "encryptionkey@0"
I20260925 22:54:48.054121 27250 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:54:48.056424 27265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:48.057690 27250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:54:48.057813 27250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "92bfd5c259204d5994cfd0029b038c21"
format_stamp: "Formatted at 2026-09-25 22:54:48 on dist-test-slave-x1lh"
server_key: "8a000aa7209f7ea6ca511865a72c311d"
server_key_iv: "ab611e92164c515cf2b3677d94ee5463"
server_key_version: "encryptionkey@0"
I20260925 22:54:48.057921 27250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:48.103183 27250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:48.106549 27250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:48.106734 27250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:48.107336 27250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:48.108130 27250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:48.108196 27250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:48.108263 27250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:48.108311 27250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:48.116990 27250 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:43103
I20260925 22:54:48.117038 27378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:43103 every 8 connection(s)
I20260925 22:54:48.117885 27250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:54:48.126739 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 27250
I20260925 22:54:48.126863 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:54:48.127134 20939 external_mini_cluster.cc:1428] Setting key a02a208d0ab5548ce07b324f8d061b37
Sep 25 22:54:48 dist-test-slave-x1lh krb5kdc[27009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376888, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM
I20260925 22:54:48.131901 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.119684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:40987 (local address 127.20.114.254:37819)
0925 22:54:48.119829 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:48.119833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:48.120638 (+ 805us) server_negotiation.cc:407] Connection header received
0925 22:54:48.122032 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:48.122035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:48.122092 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:48.122198 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:48.123322 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.123861 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.125035 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.125266 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.125380 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.127906 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:48.127910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:48.127912 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:48.127934 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:48.129769 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.130290 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.130296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.130300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.130398 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.130811 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.130813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.130814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.130986 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:48.131060 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:48.131695 (+ 635us) server_negotiation.cc:299] Negotiation successful
0925 22:54:48.131785 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:54:48.133154 27381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.120067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.20.114.194:40987)
0925 22:54:48.120494 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:48.120535 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:48.121821 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:48.122365 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:48.122371 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:48.122714 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:48.123180 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.123193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.123986 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.123991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.124853 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.124921 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.125417 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.125421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.125594 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.126124 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:48.126140 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:48.127748 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:48.129885 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.129889 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.129921 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.130190 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.130498 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.130505 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.130509 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.130716 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.131088 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:48.131097 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:48.131424 (+ 327us) client_negotiation.cc:769] Sending connection context
0925 22:54:48.131665 (+ 241us) client_negotiation.cc:240] Negotiation successful
0925 22:54:48.132004 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":101,"threads_started":1}
I20260925 22:54:48.134333 27379 heartbeater.cc:347] Connected to a master server at 127.20.114.254:37819
I20260925 22:54:48.134562 27379 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:48.135030 27379 heartbeater.cc:511] Master 127.20.114.254:37819 requested a full tablet report, sending...
I20260925 22:54:48.136047 27054 ts_manager.cc:194] Registered new tserver with Master: 92bfd5c259204d5994cfd0029b038c21 (127.20.114.194:43103)
I20260925 22:54:48.136611 27054 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:40987
WARNING: no policy specified for oryx/127.20.114.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.114.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.114.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.195.keytab.
Entry for principal oryx/127.20.114.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.195.keytab.
I20260925 22:54:48.155869 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37819
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.195.keytab
--principal=oryx/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:48.265802 27384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:48.266084 27384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:48.266160 27384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:48.270658 27384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:48.270740 27384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:48.270838 27384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:54:48.276113 27384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.195.keytab
--principal=oryx/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:48.277232 27384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:48.278327 27384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:54:48.288017 27384 server_base.cc:1034] running on GCE node
W20260925 22:54:48.287928 27390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:48.287992 27392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:48.287951 27389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:48.288580 27384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:48.289271 27384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:48.290483 27384 hybrid_clock.cc:648] HybridClock initialized: now 1790376888290443 us; error 43 us; skew 500 ppm
Sep 25 22:54:48 dist-test-slave-x1lh krb5kdc[27009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376888, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:48.293236 27384 init.cc:377] Logged in from keytab as oryx/127.20.114.195@KRBTEST.COM (short username oryx)
I20260925 22:54:48.294024 27384 webserver.cc:460] Webserver started at http://127.20.114.195:42699/ using document root <none> and password file <none>
I20260925 22:54:48.294574 27384 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:48.294661 27384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:48.294884 27384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:48.296855 27384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "a341f62c9efa4a52995a7670f9e61721"
format_stamp: "Formatted at 2026-09-25 22:54:48 on dist-test-slave-x1lh"
server_key: "71392de19e862dff0d89508e49ac7ec8"
server_key_iv: "ea52d2eb7db71b4e2515067af1a70147"
server_key_version: "encryptionkey@0"
I20260925 22:54:48.297478 27384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "a341f62c9efa4a52995a7670f9e61721"
format_stamp: "Formatted at 2026-09-25 22:54:48 on dist-test-slave-x1lh"
server_key: "71392de19e862dff0d89508e49ac7ec8"
server_key_iv: "ea52d2eb7db71b4e2515067af1a70147"
server_key_version: "encryptionkey@0"
I20260925 22:54:48.300988 27384 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:54:48.303436 27399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:48.304581 27384 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:48.304710 27384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "a341f62c9efa4a52995a7670f9e61721"
format_stamp: "Formatted at 2026-09-25 22:54:48 on dist-test-slave-x1lh"
server_key: "71392de19e862dff0d89508e49ac7ec8"
server_key_iv: "ea52d2eb7db71b4e2515067af1a70147"
server_key_version: "encryptionkey@0"
I20260925 22:54:48.304812 27384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:48.336635 27384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:48.339682 27384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:48.339884 27384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:48.340550 27384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:48.341420 27384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:48.341490 27384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:48.341558 27384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:48.341612 27384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:48.350520 27384 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:34885
I20260925 22:54:48.350545 27512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:34885 every 8 connection(s)
I20260925 22:54:48.351410 27384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:54:48.352101 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 27384
I20260925 22:54:48.352229 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:54:48.352499 20939 external_mini_cluster.cc:1428] Setting key 5b1307cbb4ac07d527a37aa4638654e2
Sep 25 22:54:48 dist-test-slave-x1lh krb5kdc[27009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376888, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM
I20260925 22:54:48.364749 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.353324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:35645 (local address 127.20.114.254:37819)
0925 22:54:48.353469 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:48.353475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:48.354565 (+ 1090us) server_negotiation.cc:407] Connection header received
0925 22:54:48.355929 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:48.355931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:48.355975 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:48.356070 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:48.357209 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.357736 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.358802 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.359008 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.359082 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.361526 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:48.361529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:48.361531 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:48.361561 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:48.363045 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.363474 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.363476 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.363477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.363563 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.363835 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.363837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.363839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.364007 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:48.364082 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:48.364517 (+ 435us) server_negotiation.cc:299] Negotiation successful
0925 22:54:48.364626 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:54:48.365782 27515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.353818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.20.114.195:35645)
0925 22:54:48.354386 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:48.354441 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:48.355741 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:48.356201 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:48.356208 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:48.356547 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:48.357084 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.357100 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.357836 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.357838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.358694 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.358702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.359136 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.359139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.359270 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.359774 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:48.359790 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:48.361375 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:48.363141 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.363145 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.363156 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.363392 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.363637 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.363639 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.363640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.363759 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.364098 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:48.364102 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:48.364322 (+ 220us) client_negotiation.cc:769] Sending connection context
0925 22:54:48.364472 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 22:54:48.364717 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":152,"threads_started":1}
I20260925 22:54:48.366845 27513 heartbeater.cc:347] Connected to a master server at 127.20.114.254:37819
I20260925 22:54:48.367074 27513 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:48.367566 27513 heartbeater.cc:511] Master 127.20.114.254:37819 requested a full tablet report, sending...
I20260925 22:54:48.368558 27054 ts_manager.cc:194] Registered new tserver with Master: a341f62c9efa4a52995a7670f9e61721 (127.20.114.195:34885)
I20260925 22:54:48.369230 27054 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:35645
I20260925 22:54:48.378605 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.114.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.114.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.114.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.253.keytab.
Entry for principal oryx/127.20.114.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.253.keytab.
I20260925 22:54:48.405851 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.253:35653
--webserver_interface=127.20.114.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.253.keytab
--principal=oryx/127.20.114.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:37819,127.20.114.253:35653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:48.515096 27518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:48.515352 27518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:48.515424 27518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:48.519522 27518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:48.519573 27518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:48.519593 27518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:48.519632 27518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:48.519663 27518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:48.525063 27518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:37819,127.20.114.253:35653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.253.keytab
--principal=oryx/127.20.114.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.253:35653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.114.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
W20260925 22:54:48.525280 27518 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.20.114.254:37819,127.20.114.253:35653'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 22:54:48.526173 27518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:48.527204 27518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:48.535856 27524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:48.535902 27523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:48.535863 27526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:48.536212 27518 server_base.cc:1034] running on GCE node
I20260925 22:54:48.536795 27518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:48.537909 27518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:48.539096 27518 hybrid_clock.cc:648] HybridClock initialized: now 1790376888539085 us; error 25 us; skew 500 ppm
Sep 25 22:54:48 dist-test-slave-x1lh krb5kdc[27009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376888, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:48.541894 27518 init.cc:377] Logged in from keytab as oryx/127.20.114.253@KRBTEST.COM (short username oryx)
I20260925 22:54:48.542825 27518 webserver.cc:460] Webserver started at http://127.20.114.253:36757/ using document root <none> and password file <none>
I20260925 22:54:48.543385 27518 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:48.543474 27518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:48.543721 27518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:48.545905 27518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/data/instance:
uuid: "88d59d46332e40e1af313320f228c2dc"
format_stamp: "Formatted at 2026-09-25 22:54:48 on dist-test-slave-x1lh"
server_key: "29eb707cf6780feddb72293719ee1f58"
server_key_iv: "ba6ad0ad2e55882d7b83544607134f4e"
server_key_version: "encryptionkey@0"
I20260925 22:54:48.546522 27518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/wal/instance:
uuid: "88d59d46332e40e1af313320f228c2dc"
format_stamp: "Formatted at 2026-09-25 22:54:48 on dist-test-slave-x1lh"
server_key: "29eb707cf6780feddb72293719ee1f58"
server_key_iv: "ba6ad0ad2e55882d7b83544607134f4e"
server_key_version: "encryptionkey@0"
I20260925 22:54:48.550231 27518 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:54:48.552610 27533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:48.553831 27518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:48.553970 27518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/wal
uuid: "88d59d46332e40e1af313320f228c2dc"
format_stamp: "Formatted at 2026-09-25 22:54:48 on dist-test-slave-x1lh"
server_key: "29eb707cf6780feddb72293719ee1f58"
server_key_iv: "ba6ad0ad2e55882d7b83544607134f4e"
server_key_version: "encryptionkey@0"
I20260925 22:54:48.554080 27518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:48.570420 27518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:48.573307 27518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:48.573513 27518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:48.581215 27518 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.253:35653
I20260925 22:54:48.581499 27585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.253:35653 every 8 connection(s)
I20260925 22:54:48.582342 27518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/data/info.pb
I20260925 22:54:48.585358 27586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:48.589171 27586 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:54:48.591826 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 27518
Sep 25 22:54:48 dist-test-slave-x1lh krb5kdc[27009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376888, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.253@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM
Sep 25 22:54:48 dist-test-slave-x1lh krb5kdc[27009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.114.253@KRBTEST.COM
I20260925 22:54:48.604118 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.591052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59116 (local address 127.20.114.254:37819)
0925 22:54:48.591198 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:48.591202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:48.592103 (+ 901us) server_negotiation.cc:407] Connection header received
0925 22:54:48.593247 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:48.593250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:48.593304 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:48.593395 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:48.594477 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.595340 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.596684 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.597039 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.597163 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.599721 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:48.599726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:48.599729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:48.599759 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:48.601733 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.602337 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.602341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.602343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.602437 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.602836 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.602840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.602841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.603131 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:48.603252 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:48.603815 (+ 563us) server_negotiation.cc:299] Negotiation successful
0925 22:54:48.603941 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:54:48.605233 27588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.591420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.0.0.1:59116)
0925 22:54:48.591942 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:48.591998 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:48.593015 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:48.593465 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:48.593472 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:48.593939 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:48.594316 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.594335 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.595465 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.595468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.596541 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.596562 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.597186 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.597189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.597325 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.597928 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:48.597954 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:48.599577 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:48.601848 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.601856 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.601863 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.602231 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.602527 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.602529 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.602531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.602718 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.603252 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:48.603257 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:48.603507 (+ 250us) client_negotiation.cc:769] Sending connection context
0925 22:54:48.603713 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 22:54:48.604027 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"spinlock_wait_cycles":1408,"thread_start_us":167,"threads_started":1}
I20260925 22:54:48.605233 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.592784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58098 (local address 127.20.114.253:35653)
0925 22:54:48.593095 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:48.593104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:48.593137 (+ 33us) server_negotiation.cc:407] Connection header received
0925 22:54:48.593351 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:48.593359 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:48.593577 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:48.593710 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:48.594563 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.595699 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.596796 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.597333 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.597408 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.600274 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:48.600282 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:48.600297 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:48.600323 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:48.602380 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.602853 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.602858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.602860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.602935 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.603192 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.603197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.603202 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.603492 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:48.603648 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:48.604208 (+ 560us) server_negotiation.cc:299] Negotiation successful
0925 22:54:48.604351 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"spinlock_wait_cycles":4480,"thread_start_us":125,"threads_started":1}
I20260925 22:54:48.606585 27586 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:54:48 dist-test-slave-x1lh krb5kdc[27009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376888, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.253@KRBTEST.COM for oryx/127.20.114.253@KRBTEST.COM
I20260925 22:54:48.614725 27588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.607071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:35653 (local address 127.0.0.1:58100)
0925 22:54:48.607234 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:48.607247 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:48.607342 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:48.607626 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:48.607628 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:48.607792 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:48.607867 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.607871 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.608633 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.608636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.609220 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.609228 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.610071 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.610074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.610143 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.610531 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:48.610542 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:48.611649 (+ 1107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:48.613412 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.613414 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.613416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.613612 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.613932 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.613934 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.613936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.614023 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.614344 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:48.614347 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:48.614417 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 22:54:48.614497 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 22:54:48.614594 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260925 22:54:48.614746 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.607133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58100 (local address 127.20.114.253:35653)
0925 22:54:48.607235 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:48.607241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:48.607307 (+ 66us) server_negotiation.cc:407] Connection header received
0925 22:54:48.607479 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:48.607482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:48.607518 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:48.607606 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:48.607964 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.608485 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.609313 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.609968 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.610047 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.611763 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:48.611765 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:48.611767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:48.611786 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:48.613301 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.613721 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.613725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.613730 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.613816 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.614109 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.614111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.614112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.614264 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:48.614335 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:48.614555 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 22:54:48.614638 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 22:54:48.617648 27586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc: Bootstrap starting.
I20260925 22:54:48.619997 27586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:48.620725 27586 log.cc:824] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:48.622659 27586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc: No bootstrap required, opened a new log
I20260925 22:54:48.625608 27586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } }
I20260925 22:54:48.625793 27586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:48.625869 27586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88d59d46332e40e1af313320f228c2dc, State: Initialized, Role: FOLLOWER
I20260925 22:54:48.626256 27586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } }
I20260925 22:54:48.626695 27592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } } }
I20260925 22:54:48.626827 27592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:54:48.627399 27586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [sys.catalog]: configured and running, proceeding with master startup.
W20260925 22:54:48.631757 27603 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:54:48.631842 27603 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:54:48.634207 27518 master_runner.cc:428] Detected that this master 88d59d46332e40e1af313320f228c2dc is joining an existing cluster
I20260925 22:54:48.634274 27518 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.114.253:35653
I20260925 22:54:48.643967 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.636376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58114 (local address 127.20.114.253:35653)
0925 22:54:48.636492 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:48.636498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:48.636512 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:54:48.636544 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:48.636547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:48.636609 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:48.636681 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:48.637322 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.638055 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.639055 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.639287 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.639388 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.640660 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:48.640665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:48.640668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:48.640696 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:48.642411 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.642830 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.642833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.642835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.642904 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.643175 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.643178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.643180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.643368 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:48.643431 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:48.643709 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 22:54:48.643807 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":55,"spinlock_wait_cycles":2304}
I20260925 22:54:48.644436 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.636068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:35653 (local address 127.0.0.1:58114)
0925 22:54:48.636343 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:48.636354 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:48.636455 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:48.636787 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:48.636790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:48.637031 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:48.637204 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.637211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.638170 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.638173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.638951 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.638959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.639394 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.639397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.639494 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.640014 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:48.640032 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:48.640526 (+ 494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:48.642519 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.642521 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.642523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.642735 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.642996 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.642998 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.643001 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.643070 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.643487 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:48.643490 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:48.643564 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 22:54:48.644142 (+ 578us) client_negotiation.cc:240] Negotiation successful
0925 22:54:48.644265 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":1,"spinlock_wait_cycles":2816,"thread_start_us":105,"threads_started":1}
I20260925 22:54:48.645035 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.636668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59126 (local address 127.20.114.254:37819)
0925 22:54:48.636789 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:48.636793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:48.637077 (+ 284us) server_negotiation.cc:407] Connection header received
0925 22:54:48.637189 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:48.637193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:48.637240 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:48.637322 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:48.637852 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.638640 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.639767 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.640092 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.640211 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.641771 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:48.641773 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:48.641776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:48.641798 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:48.643201 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.643636 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.643639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.643641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.643721 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.643985 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.643990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.643991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.644211 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:48.644302 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:48.644696 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 22:54:48.644897 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:54:48.645550 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.636662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.0.0.1:59126)
0925 22:54:48.636983 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:48.636997 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:48.637105 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:48.637360 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:48.637363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:48.637614 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:48.637732 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.637739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.638753 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.638757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.639649 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.639659 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.640209 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.640213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.640300 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.640990 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:48.641008 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:48.641441 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:48.643329 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.643332 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.643334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.643526 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.643811 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.643815 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.643817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.643898 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.644293 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:48.644296 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:48.644385 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 22:54:48.645231 (+ 846us) client_negotiation.cc:240] Negotiation successful
0925 22:54:48.645371 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"spinlock_wait_cycles":1792,"thread_start_us":79,"threads_started":1}
Sep 25 22:54:48 dist-test-slave-x1lh krb5kdc[27009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376887, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.254@KRBTEST.COM for oryx/127.20.114.253@KRBTEST.COM
I20260925 22:54:48.658215 27611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.649634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:35653 (local address 127.0.0.1:58118)
0925 22:54:48.649959 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:48.650000 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:48.650152 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:48.650442 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:48.650448 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:48.650785 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:48.650880 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.650886 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.651578 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.651581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.652212 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.652220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.652648 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.652651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.652765 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.653304 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:48.653316 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:48.654824 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:48.656396 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.656406 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.656412 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.656638 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.656976 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.656979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.656982 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.657065 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.657524 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:48.657532 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:48.657650 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 22:54:48.657830 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 22:54:48.658023 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":91,"threads_started":1}
I20260925 22:54:48.658476 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.649673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58118 (local address 127.20.114.253:35653)
0925 22:54:48.649780 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:48.649785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:48.650076 (+ 291us) server_negotiation.cc:407] Connection header received
0925 22:54:48.650273 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:48.650276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:48.650328 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:48.650409 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:48.650992 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.651464 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.652328 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.652538 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.652645 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.654961 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:48.654964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:48.654967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:48.655007 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:48.656286 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.656738 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.656743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.656744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.656824 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.657155 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.657160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.657161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.657416 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:48.657511 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:48.658110 (+ 599us) server_negotiation.cc:299] Negotiation successful
0925 22:54:48.658209 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:54:48.659818 27054 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.114.253:35653: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "88d59d46332e40e1af313320f228c2dc"
member_type: NON_VOTER
last_known_addr {
host: "127.20.114.253"
port: 35653
}
attrs {
promote: true
}
}
dest_uuid: "c96778fffbfd40f186940fb51039e3ae"
cas_config_opid_index: -1
I20260925 22:54:48.660423 27054 consensus_queue.cc:237] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: NON_VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: true } }
I20260925 22:54:48.662552 27615 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 88d59d46332e40e1af313320f228c2dc (127.20.114.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: NON_VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: true } } }
I20260925 22:54:48.663144 27560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:48.664335 27615 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.114.253:35653
I20260925 22:54:48.664412 27560 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [term 1 FOLLOWER]: Refusing update from remote peer c96778fffbfd40f186940fb51039e3ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 22:54:48.664733 27518 master.cc:537] Master@127.20.114.253:35653 shutting down...
I20260925 22:54:48.665225 27615 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c96778fffbfd40f186940fb51039e3ae" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: NON_VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: true } } }
I20260925 22:54:48.665336 27615 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:48.665743 27615 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: NON_VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 22:54:48.667582 27040 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
WARNING: no policy specified for oryx/127.20.114.252@KRBTEST.COM; defaulting to no policy
W20260925 22:54:48.669039 27040 consensus_peers.cc:487] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae -> Peer 88d59d46332e40e1af313320f228c2dc (127.20.114.253:35653): Couldn't send request to peer 88d59d46332e40e1af313320f228c2dc. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 22:54:48.669498 27518 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:48.669759 27518 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:48.669858 27518 tablet_replica.cc:330] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.114.252@KRBTEST.COM" created.
I20260925 22:54:48.685802 27518 master.cc:552] Master@127.20.114.253:35653 shutdown complete.
I20260925 22:54:48.685878 27518 master_runner.cc:305] Clearing existing system tablet
I20260925 22:54:48.686111 27518 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:48.686178 27518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:48.687497 27518 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 22:54:48.689038 27620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:48.689391 27518 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:48.689531 27518 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 22:54:48.689566 27518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/wal
uuid: "88d59d46332e40e1af313320f228c2dc"
format_stamp: "Formatted at 2026-09-25 22:54:48 on dist-test-slave-x1lh"
server_key: "29eb707cf6780feddb72293719ee1f58"
server_key_iv: "ba6ad0ad2e55882d7b83544607134f4e"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.114.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.252.keytab.
Entry for principal oryx/127.20.114.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.252.keytab.
I20260925 22:54:48.691643 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.252:45987
--webserver_interface=127.20.114.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.252.keytab
--principal=oryx/127.20.114.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:37819,127.20.114.253:35653,127.20.114.252:45987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:54:48.692131 27518 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:48.694514 27518 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 22:54:48.694614 27518 log.cc:1196] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 22:54:48.694864 27518 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc: Deleting consensus metadata
I20260925 22:54:48.695030 27518 master_runner.cc:315] Copying system tablet from 127.20.114.254:37819
I20260925 22:54:48.696202 27518 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc: tablet copy: Beginning tablet copy session from remote peer at address 127.20.114.254:37819
I20260925 22:54:48.709347 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.696930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59128 (local address 127.20.114.254:37819)
0925 22:54:48.697061 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:48.697066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:48.700644 (+ 3578us) server_negotiation.cc:407] Connection header received
0925 22:54:48.700827 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:48.700832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:48.700937 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:48.701079 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:48.701600 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.702579 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.703751 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.704116 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.704218 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.705632 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:48.705636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:48.705639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:48.705667 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:48.707370 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.707937 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.707941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.707943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.708032 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.708327 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.708330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.708332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.708598 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:48.708712 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:48.709059 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 22:54:48.709184 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:54:48.709347 27627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.696833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.0.0.1:59128)
0925 22:54:48.700504 (+ 3671us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:48.700522 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:48.700706 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:48.701115 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:48.701118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:48.701331 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:48.701484 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.701492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.702698 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.702702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.703624 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.703634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.704239 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.704243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.704343 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.705027 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:48.705043 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:48.705480 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:48.707553 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.707556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.707559 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.707851 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.708134 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.708137 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.708139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.708235 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.708700 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:48.708703 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:48.708802 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 22:54:48.708975 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 22:54:48.709122 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3583,"thread_start_us":90,"threads_started":1}
I20260925 22:54:48.710409 27074 tablet_copy_service.cc:140] P c96778fffbfd40f186940fb51039e3ae: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 88d59d46332e40e1af313320f228c2dc ({username='oryx', principal='oryx/127.20.114.253@KRBTEST.COM'} at 127.0.0.1:59128)
I20260925 22:54:48.710605 27074 tablet_copy_service.cc:161] P c96778fffbfd40f186940fb51039e3ae: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 88d59d46332e40e1af313320f228c2dc at {username='oryx', principal='oryx/127.20.114.253@KRBTEST.COM'} at 127.0.0.1:59128: session id = 88d59d46332e40e1af313320f228c2dc-00000000000000000000000000000000
I20260925 22:54:48.712468 27074 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 22:54:48.714273 27518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 22:54:48.716636 27518 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc: tablet copy: Starting download of 0 data blocks...
I20260925 22:54:48.716828 27518 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc: tablet copy: Starting download of 1 WAL segments...
I20260925 22:54:48.718657 27518 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 22:54:48.720336 27074 tablet_copy_service.cc:342] P c96778fffbfd40f186940fb51039e3ae: Request end of tablet copy session 88d59d46332e40e1af313320f228c2dc-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.114.253@KRBTEST.COM'} at 127.0.0.1:59128
I20260925 22:54:48.720459 27074 tablet_copy_service.cc:433] P c96778fffbfd40f186940fb51039e3ae: ending tablet copy session 88d59d46332e40e1af313320f228c2dc-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 88d59d46332e40e1af313320f228c2dc
I20260925 22:54:48.727902 27518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:48.729624 27631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:48.729923 27518 server_base.cc:1034] running on GCE node
W20260925 22:54:48.733348 27628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:48.733839 27629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:48.734138 27518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:48.734447 27518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:48.735589 27518 hybrid_clock.cc:648] HybridClock initialized: now 1790376888735564 us; error 40 us; skew 500 ppm
Sep 25 22:54:48 dist-test-slave-x1lh krb5kdc[27009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376888, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:48.737284 27518 init.cc:377] Logged in from keytab as oryx/127.20.114.253@KRBTEST.COM (short username oryx)
I20260925 22:54:48.737741 27518 webserver.cc:460] Webserver started at http://127.20.114.253:33815/ using document root <none> and password file <none>
I20260925 22:54:48.737928 27518 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:48.737972 27518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:48.739281 27518 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:48.740269 27638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:48.740558 27518 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:48.740641 27518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/wal
uuid: "88d59d46332e40e1af313320f228c2dc"
format_stamp: "Formatted at 2026-09-25 22:54:48 on dist-test-slave-x1lh"
server_key: "29eb707cf6780feddb72293719ee1f58"
server_key_iv: "ba6ad0ad2e55882d7b83544607134f4e"
server_key_version: "encryptionkey@0"
I20260925 22:54:48.740724 27518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:48.760011 27518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:48.762259 27518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:48.766894 27518 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.253:35653
I20260925 22:54:48.767040 27690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.253:35653 every 8 connection(s)
I20260925 22:54:48.767473 27518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-1/data/info.pb
I20260925 22:54:48.769263 27691 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:54:48.770356 27691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc: Bootstrap starting.
I20260925 22:54:48.780789 27691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:54:48.781132 27691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc: Bootstrap complete.
I20260925 22:54:48.781734 27691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: NON_VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: true } }
I20260925 22:54:48.781890 27691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 88d59d46332e40e1af313320f228c2dc, State: Initialized, Role: LEARNER
I20260925 22:54:48.782004 27691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: NON_VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: true } }
I20260925 22:54:48.782342 27694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: NON_VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: true } } }
I20260925 22:54:48.782449 27691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:48.782529 27694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [sys.catalog]: This master's current role is: LEARNER
I20260925 22:54:48.786063 27705 catalog_manager.cc:1259] Loaded cluster ID: 0caef3ffb432457ba13ec0443bf9ca8a
I20260925 22:54:48.786125 27705 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc: loading cluster ID for follower catalog manager: success
I20260925 22:54:48.788261 27705 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc: acquiring CA information for follower catalog manager: success
I20260925 22:54:48.789755 27705 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 22:54:48.823680 27621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:48.823913 27621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:48.823952 27621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:48.827870 27621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:48.827924 27621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:48.827943 27621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:48.827960 27621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:48.827975 27621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:48.833050 27621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:37819,127.20.114.253:35653,127.20.114.252:45987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.252.keytab
--principal=oryx/127.20.114.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.252:45987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.114.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:48.833988 27621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:48.834916 27621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:48.843052 27711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:48.843062 27713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:48.843050 27710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:48.843603 27621 server_base.cc:1034] running on GCE node
I20260925 22:54:48.844132 27621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:48.845325 27621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:48.846500 27621 hybrid_clock.cc:648] HybridClock initialized: now 1790376888846474 us; error 55 us; skew 500 ppm
Sep 25 22:54:48 dist-test-slave-x1lh krb5kdc[27009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376888, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:48.849170 27621 init.cc:377] Logged in from keytab as oryx/127.20.114.252@KRBTEST.COM (short username oryx)
I20260925 22:54:48.850013 27621 webserver.cc:460] Webserver started at http://127.20.114.252:35489/ using document root <none> and password file <none>
I20260925 22:54:48.850569 27621 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:48.850659 27621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:48.850884 27621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:48.852909 27621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/data/instance:
uuid: "1fc6c03407644ff6b318682cd66988a9"
format_stamp: "Formatted at 2026-09-25 22:54:48 on dist-test-slave-x1lh"
server_key: "12b1b5d269c832b0ad222ac2128760b1"
server_key_iv: "aea1b16ed6c5865fabe1d1a610c630c5"
server_key_version: "encryptionkey@0"
I20260925 22:54:48.853488 27621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/wal/instance:
uuid: "1fc6c03407644ff6b318682cd66988a9"
format_stamp: "Formatted at 2026-09-25 22:54:48 on dist-test-slave-x1lh"
server_key: "12b1b5d269c832b0ad222ac2128760b1"
server_key_iv: "aea1b16ed6c5865fabe1d1a610c630c5"
server_key_version: "encryptionkey@0"
I20260925 22:54:48.857115 27621 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:54:48.859390 27720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:48.860527 27621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:48.860663 27621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/wal
uuid: "1fc6c03407644ff6b318682cd66988a9"
format_stamp: "Formatted at 2026-09-25 22:54:48 on dist-test-slave-x1lh"
server_key: "12b1b5d269c832b0ad222ac2128760b1"
server_key_iv: "aea1b16ed6c5865fabe1d1a610c630c5"
server_key_version: "encryptionkey@0"
I20260925 22:54:48.860771 27621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:48.881990 27621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:48.885258 27621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:48.885461 27621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:48.892287 27621 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.252:45987
I20260925 22:54:48.892560 27772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.252:45987 every 8 connection(s)
I20260925 22:54:48.893343 27621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/data/info.pb
I20260925 22:54:48.896004 27773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:48.898867 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 27621
I20260925 22:54:48.898950 27245 heartbeater.cc:503] Master 127.20.114.254:37819 was elected leader, sending a full tablet report...
I20260925 22:54:48.899873 27773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:54:48 dist-test-slave-x1lh krb5kdc[27009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376888, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.252@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM
Sep 25 22:54:48 dist-test-slave-x1lh krb5kdc[27009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.114.252@KRBTEST.COM
I20260925 22:54:48.913194 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.901925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59142 (local address 127.20.114.254:37819)
0925 22:54:48.902064 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:48.902068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:48.902360 (+ 292us) server_negotiation.cc:407] Connection header received
0925 22:54:48.902508 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:48.902511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:48.902553 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:48.902603 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:48.903634 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.904138 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.905532 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.905872 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.905978 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.909165 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:48.909168 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:48.909171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:48.909199 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:48.910835 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.911443 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.911447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.911449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.911562 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.911921 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.911924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.911925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.912195 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:48.912284 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:48.912856 (+ 572us) server_negotiation.cc:299] Negotiation successful
0925 22:54:48.913006 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:54:48.914196 27775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.900436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50316 (local address 127.20.114.252:45987)
0925 22:54:48.900893 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:48.900913 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:48.900954 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:54:48.901773 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:48.901792 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:48.902225 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:48.902523 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:48.903265 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.904480 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.905631 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.906217 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.906317 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.909589 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:48.909598 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:48.909614 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:48.909634 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:48.911311 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.911822 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.911828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.911830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.911905 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.912175 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.912177 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.912178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.912628 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:48.912806 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:48.913042 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 22:54:48.913190 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":290,"spinlock_wait_cycles":3584,"thread_start_us":126,"threads_started":1}
I20260925 22:54:48.914196 27776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.901819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.0.0.1:59142)
0925 22:54:48.902225 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:48.902270 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:48.902391 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:48.902816 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:48.902820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:48.903174 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:48.903499 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.903511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.904272 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.904275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.905401 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.905411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.906037 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.906040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.906247 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.906941 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:48.906970 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:48.908997 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:48.910941 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.910951 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.910961 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.911347 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.911661 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.911663 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.911664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.911824 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.912291 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:48.912296 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:48.912628 (+ 332us) client_negotiation.cc:769] Sending connection context
0925 22:54:48.912802 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 22:54:48.913146 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":36,"spinlock_wait_cycles":1408,"thread_start_us":154,"threads_started":1}
I20260925 22:54:48.916147 27773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:54:48 dist-test-slave-x1lh krb5kdc[27009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376888, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.252@KRBTEST.COM for oryx/127.20.114.253@KRBTEST.COM
I20260925 22:54:48.923887 27776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.916619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:35653 (local address 127.0.0.1:58130)
0925 22:54:48.916732 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:48.916745 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:48.916816 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:48.917196 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:48.917198 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:48.917319 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:48.917404 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.917409 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.918225 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.918228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.918746 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.918753 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.919128 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.919131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.919198 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.919653 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:48.919671 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:48.920920 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:48.922497 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.922501 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.922503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.922692 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.922962 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.922967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.922969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.923047 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.923405 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:48.923408 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:48.923494 (+ 86us) client_negotiation.cc:769] Sending connection context
0925 22:54:48.923612 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 22:54:48.923732 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 22:54:48.923923 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.916752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58130 (local address 127.20.114.253:35653)
0925 22:54:48.917006 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:48.917010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:48.917022 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:54:48.917056 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:48.917058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:48.917097 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:48.917170 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:48.917497 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.918142 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.918838 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.919045 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.919113 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.921051 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:48.921055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:48.921057 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:48.921083 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:48.922386 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.922790 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.922795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.922796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.922874 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.923144 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.923147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.923149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.923315 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:48.923415 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:48.923662 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 22:54:48.923770 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":120,"threads_started":1}
I20260925 22:54:48.924606 27773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:54:48 dist-test-slave-x1lh krb5kdc[27009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376888, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.252@KRBTEST.COM for oryx/127.20.114.252@KRBTEST.COM
I20260925 22:54:48.931207 27776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.925068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:45987 (local address 127.0.0.1:50332)
0925 22:54:48.925182 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:48.925196 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:48.925263 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:48.925417 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:48.925419 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:48.925530 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:48.925604 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.925608 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.926246 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.926249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.926724 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.926731 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.927112 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.927114 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.927165 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.927505 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:48.927515 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:48.928549 (+ 1034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:48.930148 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.930150 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.930152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.930316 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.930531 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.930533 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.930534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.930604 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.930889 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:48.930891 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:48.930952 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 22:54:48.931023 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 22:54:48.931103 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 22:54:48.931288 27775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.925129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50332 (local address 127.20.114.252:45987)
0925 22:54:48.925237 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:48.925241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:48.925260 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:54:48.925319 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:48.925321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:48.925351 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:48.925404 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:48.925695 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.926156 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.926810 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.927027 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.927096 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.928669 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:48.928672 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:48.928674 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:48.928693 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:48.930063 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.930393 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.930395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.930396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.930460 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.930675 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.930677 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.930678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.930817 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:48.930885 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:48.931082 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 22:54:48.931164 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:54:48.934514 27773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9: Bootstrap starting.
I20260925 22:54:48.936728 27773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:48.937495 27773 log.cc:824] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:48.939383 27773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9: No bootstrap required, opened a new log
I20260925 22:54:48.942390 27773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } }
I20260925 22:54:48.942591 27773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:48.942667 27773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fc6c03407644ff6b318682cd66988a9, State: Initialized, Role: FOLLOWER
I20260925 22:54:48.943075 27773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } }
I20260925 22:54:48.943527 27779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } } }
I20260925 22:54:48.943671 27779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:54:48.944267 27773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 22:54:48.949133 27790 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:54:48.949213 27790 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:54:48.952158 27621 master_runner.cc:428] Detected that this master 1fc6c03407644ff6b318682cd66988a9 is joining an existing cluster
I20260925 22:54:48.952220 27621 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.114.252:45987
I20260925 22:54:48.963977 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.954603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58132 (local address 127.20.114.253:35653)
0925 22:54:48.954717 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:48.954722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:48.955484 (+ 762us) server_negotiation.cc:407] Connection header received
0925 22:54:48.955674 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:48.955677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:48.955725 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:48.955772 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:48.956587 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.957364 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.958646 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.959035 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.959109 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.960615 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:48.960618 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:48.960621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:48.960645 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:48.962248 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.962754 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.962758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.962760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.962847 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.963125 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.963130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.963131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.963333 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:48.963439 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:48.963703 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 22:54:48.963828 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:54:48.963997 27796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.954440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:35653 (local address 127.0.0.1:58132)
0925 22:54:48.955410 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:48.955424 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:48.955560 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:48.955886 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:48.955888 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:48.956211 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:48.956445 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.956453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.957484 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.957489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.958518 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.958528 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.959267 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.959270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.959495 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.960084 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:48.960105 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:48.960456 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:48.962388 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.962392 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.962394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.962650 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.962944 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.962947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.962949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.963035 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.963427 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:48.963430 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:48.963533 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 22:54:48.963650 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 22:54:48.963800 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":181,"thread_start_us":137,"threads_started":1}
I20260925 22:54:48.967613 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 27116
I20260925 22:54:48.974125 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:33393
--local_ip_for_outbound_sockets=127.20.114.193
--tserver_master_addrs=127.20.114.254:37819
--webserver_port=40751
--webserver_interface=127.20.114.193
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.193.keytab
--principal=oryx/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.114.254:37819,127.20.114.253:35653
--tserver_master_addrs=127.20.114.254:37819,127.20.114.253:35653,127.20.114.252:45987 with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:54:48.981940 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.954534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59156 (local address 127.20.114.254:37819)
0925 22:54:48.954686 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:48.954691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:48.955224 (+ 533us) server_negotiation.cc:407] Connection header received
0925 22:54:48.955425 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:48.955428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:48.955481 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:48.955586 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:48.956273 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.957124 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.958931 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.959279 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.959356 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.978337 (+ 18981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:48.978342 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:48.978345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:48.978377 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:48.980026 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.980570 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.980574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.980575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.980667 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.980984 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.980988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.980989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.981219 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:48.981290 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:48.981640 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 22:54:48.981766 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:54:48.983414 27798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.955109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:45987 (local address 127.0.0.1:50340)
0925 22:54:48.955496 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:48.955509 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:48.955601 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:48.955885 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:48.955888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:48.956210 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:48.956449 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.956454 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.957462 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.957466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.958522 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.958528 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.959267 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.959270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.959500 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.977564 (+ 18064us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:48.977587 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:48.978215 (+ 628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:48.980490 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.980495 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.980500 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.980764 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.981111 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.981116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.981120 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.981221 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.981710 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:48.981716 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:48.981819 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 22:54:48.981931 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 22:54:48.983171 (+ 1240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":286,"spinlock_wait_cycles":5120,"thread_start_us":82,"threads_started":1}
I20260925 22:54:48.983713 27775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.955159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50340 (local address 127.20.114.252:45987)
0925 22:54:48.955276 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:48.955281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:48.955557 (+ 276us) server_negotiation.cc:407] Connection header received
0925 22:54:48.955686 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:48.955689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:48.955721 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:48.955766 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:48.956563 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.957337 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.958638 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.959031 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.959109 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.978388 (+ 19279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:48.978394 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:48.978399 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:48.978426 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:48.980360 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.980901 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.980907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.980910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.980991 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.981323 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.981328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.981332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.981569 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:48.981672 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:48.983413 (+ 1741us) server_negotiation.cc:299] Negotiation successful
0925 22:54:48.983537 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:54:48.984237 27797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.954437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.0.0.1:59156)
0925 22:54:48.955028 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:48.955056 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:48.955306 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:48.955708 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:48.955712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:48.955995 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:48.956133 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.956141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.957261 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.957264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.958768 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.958779 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.959501 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.959505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.959656 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.977502 (+ 17846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:48.977522 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:48.978114 (+ 592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:48.980160 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.980163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.980165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.980455 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.980774 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.980776 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.980778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.980855 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.981357 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:48.981360 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:48.981454 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 22:54:48.982528 (+ 1074us) client_negotiation.cc:240] Negotiation successful
0925 22:54:48.983286 (+ 758us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":532,"thread_start_us":95,"threads_started":1}
Sep 25 22:54:48 dist-test-slave-x1lh krb5kdc[27009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376887, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.254@KRBTEST.COM for oryx/127.20.114.252@KRBTEST.COM
I20260925 22:54:48.997673 27611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.988442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:45987 (local address 127.0.0.1:50348)
0925 22:54:48.988583 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:48.988599 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:48.988677 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:48.989056 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:48.989060 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:48.989255 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:48.989360 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.989367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.990391 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.990395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.991279 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:48.991289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.991866 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.991869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:48.991967 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.992462 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:48.992476 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:48.994034 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:48.996049 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.996053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.996055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.996313 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.996602 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:48.996605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:48.996607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:48.996698 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:48.997127 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:48.997130 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:48.997232 (+ 102us) client_negotiation.cc:769] Sending connection context
0925 22:54:48.997340 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 22:54:48.997484 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 22:54:48.997684 27775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:48.988659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50348 (local address 127.20.114.252:45987)
0925 22:54:48.988752 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:48.988756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:48.988767 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:54:48.988834 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:48.988836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:48.988937 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:48.989015 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:48.989487 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.990281 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.991397 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:48.991753 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:48.991852 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:48.994196 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:48.994200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:48.994203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:48.994242 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:48.995937 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.996415 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.996418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.996420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.996503 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:48.996790 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:48.996793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:48.996795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:48.997024 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:48.997127 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:48.997415 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 22:54:48.997530 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 22:54:48.999097 27054 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.114.252:45987: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1fc6c03407644ff6b318682cd66988a9"
member_type: NON_VOTER
last_known_addr {
host: "127.20.114.252"
port: 45987
}
attrs {
promote: true
}
}
dest_uuid: "c96778fffbfd40f186940fb51039e3ae"
cas_config_opid_index: 5
I20260925 22:54:48.999740 27054 consensus_queue.cc:237] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: NON_VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: true } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: NON_VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: true } }
I20260925 22:54:49.001711 27747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:49.003214 27747 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [term 1 FOLLOWER]: Refusing update from remote peer c96778fffbfd40f186940fb51039e3ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 22:54:49.003783 27614 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: NON_VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:49.000819 27615 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1fc6c03407644ff6b318682cd66988a9 (127.20.114.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: NON_VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: true } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: NON_VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: true } } }
I20260925 22:54:49.005442 27615 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.114.252:45987
I20260925 22:54:49.005631 27615 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c96778fffbfd40f186940fb51039e3ae" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: NON_VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: true } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: NON_VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: true } } }
I20260925 22:54:49.005733 27615 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:49.006101 27621 master.cc:537] Master@127.20.114.252:45987 shutting down...
I20260925 22:54:49.009013 27747 raft_consensus.cc:858] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: NON_VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: NON_VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: true } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: NON_VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: true } }
W20260925 22:54:49.009141 27747 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: NON_VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: NON_VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: true } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: NON_VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: true } }
I20260925 22:54:49.010555 27747 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 88d59d46332e40e1af313320f228c2dc (127.20.114.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: NON_VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: true } } }
I20260925 22:54:49.012153 27779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c96778fffbfd40f186940fb51039e3ae. Latest consensus state: current_term: 1 leader_uuid: "c96778fffbfd40f186940fb51039e3ae" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: NON_VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: true } } }
I20260925 22:54:49.012254 27779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 22:54:49.012327 27779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c96778fffbfd40f186940fb51039e3ae" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: NON_VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: true } } }
I20260925 22:54:49.012382 27779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 22:54:49.012434 27779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c96778fffbfd40f186940fb51039e3ae" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: NON_VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: true } } }
I20260925 22:54:49.012480 27779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 22:54:49.016997 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.003630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58134 (local address 127.20.114.253:35653)
0925 22:54:49.003751 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:49.003755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:49.003769 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:54:49.003818 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:49.003820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:49.003884 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:49.003978 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:49.012907 (+ 8929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.013700 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.015563 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.016187 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.016285 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.016406 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:49.016772 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 22:54:49.016850 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:54:49.017423 27611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.001200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:35653 (local address 127.0.0.1:58134)
0925 22:54:49.001326 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:49.001342 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:49.001416 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:49.012596 (+ 11180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:49.012599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:49.012624 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:49.012725 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.012732 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.013942 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.013945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.015403 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.015415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.016508 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.016512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.016606 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.016652 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 22:54:49.017205 (+ 553us) client_negotiation.cc:240] Negotiation successful
0925 22:54:49.017297 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 22:54:49.017974 27665 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [term 1 LEARNER]: Refusing update from remote peer c96778fffbfd40f186940fb51039e3ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 22:54:49.018379 27800 mvcc.cc:205] Tried to move back new op lower bound from 7333383731842199552 to 7333383731609374720. Current Snapshot: MvccSnapshot[applied={T|T < 7333383731642425344}]
I20260925 22:54:49.018469 27614 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: NON_VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 22:54:49.019534 27614 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c96778fffbfd40f186940fb51039e3ae" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: NON_VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: true } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: NON_VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: true } } }
I20260925 22:54:49.019651 27614 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:49.020279 27665 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1fc6c03407644ff6b318682cd66988a9 (127.20.114.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: NON_VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: true } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: NON_VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: true } } }
W20260925 22:54:49.020910 27041 consensus_peers.cc:487] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae -> Peer 1fc6c03407644ff6b318682cd66988a9 (127.20.114.252:45987): Couldn't send request to peer 1fc6c03407644ff6b318682cd66988a9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 22:54:49.022204 27621 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260925 22:54:49.022415 27621 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260925 22:54:49.022480 27621 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9: stopping tablet replica
I20260925 22:54:49.022571 27694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c96778fffbfd40f186940fb51039e3ae. Latest consensus state: current_term: 1 leader_uuid: "c96778fffbfd40f186940fb51039e3ae" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: NON_VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: true } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: NON_VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: true } } }
I20260925 22:54:49.022678 27694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [sys.catalog]: This master's current role is: LEARNER
I20260925 22:54:49.022763 27694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c96778fffbfd40f186940fb51039e3ae" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: NON_VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: true } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: NON_VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: true } } }
I20260925 22:54:49.022822 27694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [sys.catalog]: This master's current role is: LEARNER
I20260925 22:54:49.026647 27615 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae: attempting to promote NON_VOTER 88d59d46332e40e1af313320f228c2dc to VOTER
I20260925 22:54:49.027047 27615 consensus_queue.cc:237] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: NON_VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: true } }
I20260925 22:54:49.028837 27611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.028550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:45987 (local address 127.0.0.1:50364)
0925 22:54:49.028686 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:49.028759 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:45987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 22:54:49.029561 27665 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [term 1 LEARNER]: Refusing update from remote peer c96778fffbfd40f186940fb51039e3ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 22:54:49.029645 27611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.029402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:45987 (local address 127.0.0.1:50366)
0925 22:54:49.029524 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:49.029577 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:45987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 22:54:49.029760 27041 consensus_peers.cc:487] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae -> Peer 1fc6c03407644ff6b318682cd66988a9 (127.20.114.252:45987): Couldn't send request to peer 1fc6c03407644ff6b318682cd66988a9. Status: Network error: Client connection negotiation failed: client connection to 127.20.114.252:45987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 22:54:49.030216 27615 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 22:54:49.031984 27614 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 88d59d46332e40e1af313320f228c2dc (127.20.114.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: NON_VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: true } } }
I20260925 22:54:49.032649 27665 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 88d59d46332e40e1af313320f228c2dc (127.20.114.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: NON_VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: true } } }
I20260925 22:54:49.033475 27614 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c96778fffbfd40f186940fb51039e3ae" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: NON_VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: true } } }
I20260925 22:54:49.033581 27614 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:49.033721 27614 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c96778fffbfd40f186940fb51039e3ae" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: NON_VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: true } } }
I20260925 22:54:49.033804 27614 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:49.034415 27808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c96778fffbfd40f186940fb51039e3ae" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: NON_VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: true } } }
I20260925 22:54:49.034526 27808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:54:49.042977 27621 master.cc:552] Master@127.20.114.252:45987 shutdown complete.
I20260925 22:54:49.043100 27621 master_runner.cc:305] Clearing existing system tablet
I20260925 22:54:49.043388 27621 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:49.043476 27621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:49.044996 27621 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:49.046137 27811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:49.046416 27621 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:49.046519 27621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:49.046550 27621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/wal
uuid: "1fc6c03407644ff6b318682cd66988a9"
format_stamp: "Formatted at 2026-09-25 22:54:48 on dist-test-slave-x1lh"
server_key: "12b1b5d269c832b0ad222ac2128760b1"
server_key_iv: "aea1b16ed6c5865fabe1d1a610c630c5"
server_key_version: "encryptionkey@0"
I20260925 22:54:49.048267 27621 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:49.050614 27621 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 22:54:49.050698 27621 log.cc:1196] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 22:54:49.050974 27621 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9: Deleting consensus metadata
I20260925 22:54:49.051132 27621 master_runner.cc:315] Copying system tablet from 127.20.114.254:37819
I20260925 22:54:49.052241 27621 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.114.254:37819
I20260925 22:54:49.061213 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.052892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59162 (local address 127.20.114.254:37819)
0925 22:54:49.053019 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:49.053024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:49.053250 (+ 226us) server_negotiation.cc:407] Connection header received
0925 22:54:49.053395 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:49.053399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:49.053451 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:49.053532 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:49.054022 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.054836 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.055951 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.056289 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.056392 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.057701 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:49.057705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:49.057707 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:49.057733 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:49.059417 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.059897 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.059900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.059902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.059990 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.060289 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.060292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.060294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.060515 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:49.060625 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:49.060932 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 22:54:49.061051 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:54:49.061625 27817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.052766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.0.0.1:59162)
0925 22:54:49.053141 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:49.053155 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:49.053286 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:49.053567 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:49.053571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:49.053784 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:49.053898 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.053904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.054960 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.054964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.055821 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.055832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.056414 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.056418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.056505 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.057100 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:49.057116 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:49.057555 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:49.059533 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:49.059537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:49.059539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:49.059786 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:49.060096 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:49.060099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:49.060101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:49.060190 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:49.060618 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:49.060621 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:49.060721 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 22:54:49.060829 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 22:54:49.061093 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":135,"threads_started":1}
I20260925 22:54:49.062011 27074 tablet_copy_service.cc:140] P c96778fffbfd40f186940fb51039e3ae: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1fc6c03407644ff6b318682cd66988a9 ({username='oryx', principal='oryx/127.20.114.252@KRBTEST.COM'} at 127.0.0.1:59162)
I20260925 22:54:49.062120 27074 tablet_copy_service.cc:161] P c96778fffbfd40f186940fb51039e3ae: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1fc6c03407644ff6b318682cd66988a9 at {username='oryx', principal='oryx/127.20.114.252@KRBTEST.COM'} at 127.0.0.1:59162: session id = 1fc6c03407644ff6b318682cd66988a9-00000000000000000000000000000000
I20260925 22:54:49.063169 27074 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 22:54:49.064388 27621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 22:54:49.066654 27621 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9: tablet copy: Starting download of 0 data blocks...
I20260925 22:54:49.066905 27621 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9: tablet copy: Starting download of 1 WAL segments...
I20260925 22:54:49.068500 27621 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 22:54:49.070289 27074 tablet_copy_service.cc:342] P c96778fffbfd40f186940fb51039e3ae: Request end of tablet copy session 1fc6c03407644ff6b318682cd66988a9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.114.252@KRBTEST.COM'} at 127.0.0.1:59162
I20260925 22:54:49.070371 27074 tablet_copy_service.cc:433] P c96778fffbfd40f186940fb51039e3ae: ending tablet copy session 1fc6c03407644ff6b318682cd66988a9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1fc6c03407644ff6b318682cd66988a9
I20260925 22:54:49.077863 27621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:49.079492 27821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:49.079788 27818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:49.079941 27621 server_base.cc:1034] running on GCE node
W20260925 22:54:49.080279 27819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:49.080571 27621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:49.080914 27621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:49.082090 27621 hybrid_clock.cc:648] HybridClock initialized: now 1790376889082076 us; error 27 us; skew 500 ppm
Sep 25 22:54:49 dist-test-slave-x1lh krb5kdc[27009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376889, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:49.083943 27621 init.cc:377] Logged in from keytab as oryx/127.20.114.252@KRBTEST.COM (short username oryx)
I20260925 22:54:49.084398 27621 webserver.cc:460] Webserver started at http://127.20.114.252:42755/ using document root <none> and password file <none>
I20260925 22:54:49.084609 27621 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:49.084656 27621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:49.086036 27621 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 22:54:49.087090 27828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:49.087369 27621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 22:54:49.087442 27621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/wal
uuid: "1fc6c03407644ff6b318682cd66988a9"
format_stamp: "Formatted at 2026-09-25 22:54:48 on dist-test-slave-x1lh"
server_key: "12b1b5d269c832b0ad222ac2128760b1"
server_key_iv: "aea1b16ed6c5865fabe1d1a610c630c5"
server_key_version: "encryptionkey@0"
I20260925 22:54:49.087525 27621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:49.109532 27621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:49.111701 27621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:49.116561 27621 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.252:45987
I20260925 22:54:49.117276 27621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/master-2/data/info.pb
I20260925 22:54:49.117305 27880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.252:45987 every 8 connection(s)
W20260925 22:54:49.126693 27799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:49.127080 27799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:49.127228 27799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 22:54:49.128835 27881 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:54:49.129967 27881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9: Bootstrap starting.
W20260925 22:54:49.133415 27799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:49.133507 27799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:49.133633 27799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:54:49.137712 27881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:54:49.138005 27881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9: Bootstrap complete.
I20260925 22:54:49.138604 27379 heartbeater.cc:503] Master 127.20.114.254:37819 was elected leader, sending a full tablet report...
I20260925 22:54:49.138592 27881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: NON_VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: true } }
I20260925 22:54:49.138725 27881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fc6c03407644ff6b318682cd66988a9, State: Initialized, Role: LEARNER
I20260925 22:54:49.138841 27881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: NON_VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: true } }
I20260925 22:54:49.139312 27881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:49.139423 27887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: NON_VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: true } } }
I20260925 22:54:49.139614 27887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [sys.catalog]: This master's current role is: LEARNER
I20260925 22:54:49.139638 27799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.193.keytab
--principal=oryx/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:33393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=40751
--tserver_master_addrs=127.20.114.254:37819,127.20.114.253:35653,127.20.114.252:45987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:49.140838 27799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:49.141989 27898 catalog_manager.cc:1259] Loaded cluster ID: 0caef3ffb432457ba13ec0443bf9ca8a
I20260925 22:54:49.142030 27898 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9: loading cluster ID for follower catalog manager: success
I20260925 22:54:49.142143 27799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:54:49.143565 27898 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9: acquiring CA information for follower catalog manager: success
I20260925 22:54:49.144661 27898 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 22:54:49.152501 27901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:49.152501 27900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:49.152535 27799 server_base.cc:1034] running on GCE node
W20260925 22:54:49.152537 27903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:49.153119 27799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:49.153728 27799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:49.154886 27799 hybrid_clock.cc:648] HybridClock initialized: now 1790376889154867 us; error 29 us; skew 500 ppm
Sep 25 22:54:49 dist-test-slave-x1lh krb5kdc[27009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376889, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:49.157603 27799 init.cc:377] Logged in from keytab as oryx/127.20.114.193@KRBTEST.COM (short username oryx)
I20260925 22:54:49.158357 27799 webserver.cc:460] Webserver started at http://127.20.114.193:40751/ using document root <none> and password file <none>
I20260925 22:54:49.158833 27799 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:49.158917 27799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:49.162643 27799 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:54:49.164459 27910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:49.165618 27799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:49.165758 27799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "c01ee4ba045949eebcc2660e769f7b6f"
format_stamp: "Formatted at 2026-09-25 22:54:47 on dist-test-slave-x1lh"
server_key: "86cdc06759ab8f71d89b53d01f48d025"
server_key_iv: "5366ff965476c4bec1502a6116c858e6"
server_key_version: "encryptionkey@0"
I20260925 22:54:49.166267 27799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:49.181022 27799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:49.184346 27799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:49.184587 27799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:49.185341 27799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:49.186439 27799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:49.186497 27799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:49.186547 27799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:49.186575 27799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:49.195901 27799 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:33393
I20260925 22:54:49.195920 28023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:33393 every 8 connection(s)
I20260925 22:54:49.196918 27799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:54:49.203279 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 27799
I20260925 22:54:49.203481 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 27250
Sep 25 22:54:49 dist-test-slave-x1lh krb5kdc[27009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376889, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM
Sep 25 22:54:49 dist-test-slave-x1lh krb5kdc[27009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376889, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.252@KRBTEST.COM
Sep 25 22:54:49 dist-test-slave-x1lh krb5kdc[27009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376889, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.253@KRBTEST.COM
I20260925 22:54:49.213850 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:43103
--local_ip_for_outbound_sockets=127.20.114.194
--tserver_master_addrs=127.20.114.254:37819
--webserver_port=34629
--webserver_interface=127.20.114.194
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.194.keytab
--principal=oryx/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.114.254:37819,127.20.114.253:35653
--tserver_master_addrs=127.20.114.254:37819,127.20.114.253:35653,127.20.114.252:45987 with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:54:49.214943 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.198595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:58211 (local address 127.20.114.253:35653)
0925 22:54:49.198775 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:49.198783 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:49.199659 (+ 876us) server_negotiation.cc:407] Connection header received
0925 22:54:49.200952 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:49.200956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:49.201016 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:49.201092 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:49.202424 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.203263 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.205274 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.205632 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.205699 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.210548 (+ 4849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:49.210552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:49.210555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:49.210588 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:49.212378 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.212957 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.212961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.212963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.213067 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.213569 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.213572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.213574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.213787 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:49.213868 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:49.214642 (+ 774us) server_negotiation.cc:299] Negotiation successful
0925 22:54:49.214774 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:54:49.214993 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.200151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:51507 (local address 127.20.114.254:37819)
0925 22:54:49.200271 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:49.200281 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:49.200321 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:54:49.200985 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:49.200988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:49.201038 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:49.201095 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:49.202474 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.203261 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.205275 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.205624 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.205699 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.209361 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:49.209364 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:49.209366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:49.209390 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:49.212217 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.212957 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.212961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.212963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.213180 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.213579 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.213583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.213584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.213787 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:49.213868 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:49.214642 (+ 774us) server_negotiation.cc:299] Negotiation successful
0925 22:54:49.214774 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:54:49.215060 28028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.198737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:34731 (local address 127.20.114.252:45987)
0925 22:54:49.198986 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:49.198991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:49.199660 (+ 669us) server_negotiation.cc:407] Connection header received
0925 22:54:49.200984 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:49.200988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:49.201038 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:49.201099 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:49.202441 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.203267 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.205244 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.205701 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.205792 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.210941 (+ 5149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:49.210948 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:49.210954 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:49.211001 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:49.212778 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.213359 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.213365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.213370 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.213457 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.213751 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.213757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.213762 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.213989 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:49.214076 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:49.214612 (+ 536us) server_negotiation.cc:299] Negotiation successful
0925 22:54:49.214781 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":83,"threads_started":1}
I20260925 22:54:49.216423 28029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.199058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:35653 (local address 127.20.114.193:58211)
0925 22:54:49.199510 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:49.199563 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:49.200728 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:49.201256 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:49.201262 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:49.201745 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:49.202285 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.202298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.203430 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.203435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.205141 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.205150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.205892 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.205896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.206229 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.207048 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:49.207060 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:49.210413 (+ 3353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:49.212486 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:49.212489 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:49.212491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:49.212814 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:49.213305 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:49.213310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:49.213314 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:49.213457 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:49.213939 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:49.213947 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:49.214325 (+ 378us) client_negotiation.cc:769] Sending connection context
0925 22:54:49.215107 (+ 782us) client_negotiation.cc:240] Negotiation successful
0925 22:54:49.215559 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":708,"spinlock_wait_cycles":55552,"thread_start_us":92,"threads_started":1}
I20260925 22:54:49.216722 28030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.199059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:45987 (local address 127.20.114.193:34731)
0925 22:54:49.199510 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:49.199563 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:49.200729 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:49.201255 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:49.201264 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:49.201748 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:49.202297 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.202305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.203429 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.203433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.205107 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.205121 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.205987 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.205994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.206183 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.207125 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:49.207145 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:49.210791 (+ 3646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:49.212975 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:49.212982 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:49.212987 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:49.213248 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:49.213564 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:49.213566 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:49.213568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:49.213656 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:49.214136 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:49.214141 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:49.214314 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 22:54:49.215196 (+ 882us) client_negotiation.cc:240] Negotiation successful
0925 22:54:49.215653 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":352,"spinlock_wait_cycles":25088,"thread_start_us":140,"threads_started":1}
I20260925 22:54:49.217289 28031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.200019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.20.114.193:51507)
0925 22:54:49.200251 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:49.200263 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:49.200729 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:49.201255 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:49.201264 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:49.201841 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:49.202297 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.202305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.203457 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.203461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.205139 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.205150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.205892 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.205896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.206235 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.207013 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:49.207036 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:49.209190 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:49.212353 (+ 3163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:49.212358 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:49.212373 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:49.212814 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:49.213272 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:49.213275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:49.213277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:49.213457 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:49.214163 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:49.214166 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:49.214329 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 22:54:49.216978 (+ 2649us) client_negotiation.cc:240] Negotiation successful
0925 22:54:49.217124 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":772,"spinlock_wait_cycles":27136,"thread_start_us":88,"threads_started":1}
I20260925 22:54:49.217846 28026 heartbeater.cc:347] Connected to a master server at 127.20.114.253:35653
I20260925 22:54:49.218071 28026 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:49.218364 28025 heartbeater.cc:347] Connected to a master server at 127.20.114.254:37819
I20260925 22:54:49.218443 28025 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:49.218664 28026 heartbeater.cc:511] Master 127.20.114.253:35653 requested a full tablet report, sending...
I20260925 22:54:49.218664 28025 heartbeater.cc:511] Master 127.20.114.254:37819 requested a full tablet report, sending...
I20260925 22:54:49.219211 28024 heartbeater.cc:347] Connected to a master server at 127.20.114.252:45987
I20260925 22:54:49.219290 28024 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:49.219429 28024 heartbeater.cc:511] Master 127.20.114.252:45987 requested a full tablet report, sending...
I20260925 22:54:49.219959 27054 ts_manager.cc:194] Re-registered known tserver with Master: c01ee4ba045949eebcc2660e769f7b6f (127.20.114.193:33393)
I20260925 22:54:49.220535 27054 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:51507
I20260925 22:54:49.220530 27841 ts_manager.cc:194] Registered new tserver with Master: c01ee4ba045949eebcc2660e769f7b6f (127.20.114.193:33393)
I20260925 22:54:49.221503 27647 ts_manager.cc:194] Registered new tserver with Master: c01ee4ba045949eebcc2660e769f7b6f (127.20.114.193:33393)
W20260925 22:54:49.330124 28032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:49.330366 28032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:49.330407 28032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:49.334373 28032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:49.334422 28032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:49.334484 28032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:54:49.339604 28032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.194.keytab
--principal=oryx/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:43103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=34629
--tserver_master_addrs=127.20.114.254:37819,127.20.114.253:35653,127.20.114.252:45987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:49.340749 28032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:49.341835 28032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:49.351413 28040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:49.351434 28038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:49.351434 28037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:49.351841 28032 server_base.cc:1034] running on GCE node
I20260925 22:54:49.352213 28032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:49.352787 28032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:49.353998 28032 hybrid_clock.cc:648] HybridClock initialized: now 1790376889353976 us; error 38 us; skew 500 ppm
Sep 25 22:54:49 dist-test-slave-x1lh krb5kdc[27009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376889, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:49.356583 28032 init.cc:377] Logged in from keytab as oryx/127.20.114.194@KRBTEST.COM (short username oryx)
I20260925 22:54:49.357439 28032 webserver.cc:460] Webserver started at http://127.20.114.194:34629/ using document root <none> and password file <none>
I20260925 22:54:49.357964 28032 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:49.358054 28032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:49.362212 28032 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:54:49.364789 28047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:49.366453 28032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:54:49.366600 28032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "92bfd5c259204d5994cfd0029b038c21"
format_stamp: "Formatted at 2026-09-25 22:54:48 on dist-test-slave-x1lh"
server_key: "8a000aa7209f7ea6ca511865a72c311d"
server_key_iv: "ab611e92164c515cf2b3677d94ee5463"
server_key_version: "encryptionkey@0"
I20260925 22:54:49.367419 28032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:49.371114 27513 heartbeater.cc:503] Master 127.20.114.254:37819 was elected leader, sending a full tablet report...
I20260925 22:54:49.383448 28032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:49.386349 28032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:49.386555 28032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:49.387180 28032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:49.388015 28032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:49.388059 28032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:49.388127 28032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:49.388155 28032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:49.396982 28032 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:43103
I20260925 22:54:49.397001 28160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:43103 every 8 connection(s)
I20260925 22:54:49.397957 28032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:54:49.405011 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 28032
I20260925 22:54:49.405196 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 27384
Sep 25 22:54:49 dist-test-slave-x1lh krb5kdc[27009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376889, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.253@KRBTEST.COM
Sep 25 22:54:49 dist-test-slave-x1lh krb5kdc[27009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376889, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.252@KRBTEST.COM
I20260925 22:54:49.412305 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:34885
--local_ip_for_outbound_sockets=127.20.114.195
--tserver_master_addrs=127.20.114.254:37819
--webserver_port=42699
--webserver_interface=127.20.114.195
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.195.keytab
--principal=oryx/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.114.254:37819,127.20.114.253:35653
--tserver_master_addrs=127.20.114.254:37819,127.20.114.253:35653,127.20.114.252:45987 with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:54:49 dist-test-slave-x1lh krb5kdc[27009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376889, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM
I20260925 22:54:49.414769 28028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.401464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:58297 (local address 127.20.114.252:45987)
0925 22:54:49.401588 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:49.401594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:49.401641 (+ 47us) server_negotiation.cc:407] Connection header received
0925 22:54:49.402155 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:49.402158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:49.402234 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:49.402292 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:49.403792 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.404334 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.406406 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.406832 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.406928 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.410489 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:49.410495 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:49.410502 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:49.410536 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:49.412199 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.412948 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.412957 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.412960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.413048 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.413401 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.413404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.413406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.413611 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:49.413696 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:49.414430 (+ 734us) server_negotiation.cc:299] Negotiation successful
0925 22:54:49.414579 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:54:49.415597 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.399782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:38991 (local address 127.20.114.253:35653)
0925 22:54:49.399936 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:49.399942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:49.400891 (+ 949us) server_negotiation.cc:407] Connection header received
0925 22:54:49.402143 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:49.402147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:49.402207 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:49.402302 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:49.403796 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.404601 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.407319 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.407553 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.407659 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.410177 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:49.410181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:49.410183 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:49.410212 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:49.412041 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.412703 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.412710 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.412714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.412808 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.413301 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.413307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.413312 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.413591 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:49.413681 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:49.415118 (+ 1437us) server_negotiation.cc:299] Negotiation successful
0925 22:54:49.415424 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:54:49.416545 28165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.400245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:35653 (local address 127.20.114.194:38991)
0925 22:54:49.400706 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:49.400759 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:49.401928 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:49.402511 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:49.402522 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:49.403044 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:49.403626 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.403645 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.404775 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.404778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.407184 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.407193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.407668 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.407672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.407764 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.408263 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:49.408279 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:49.409957 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:49.412181 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:49.412190 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:49.412208 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:49.412581 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:49.412945 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:49.412951 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:49.412956 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:49.413188 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:49.414810 (+ 1622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:49.414813 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:49.414903 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 22:54:49.415797 (+ 894us) client_negotiation.cc:240] Negotiation successful
0925 22:54:49.415942 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":76,"spinlock_wait_cycles":11392,"thread_start_us":102,"threads_started":1}
I20260925 22:54:49.416785 28167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.401324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:45987 (local address 127.20.114.194:58297)
0925 22:54:49.401574 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:49.401587 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:49.401929 (+ 342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:49.402522 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:49.402528 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:49.403029 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:49.403635 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.403647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.404465 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.404468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.406270 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.406283 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.407152 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.407159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.407389 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.408278 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:49.408299 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:49.410339 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:49.412304 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:49.412307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:49.412309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:49.412802 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:49.413186 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:49.413189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:49.413192 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:49.413291 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:49.413771 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:49.413779 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:49.414132 (+ 353us) client_negotiation.cc:769] Sending connection context
0925 22:54:49.415017 (+ 885us) client_negotiation.cc:240] Negotiation successful
0925 22:54:49.415415 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":76,"spinlock_wait_cycles":43264,"thread_start_us":105,"threads_started":1}
I20260925 22:54:49.418254 28161 heartbeater.cc:347] Connected to a master server at 127.20.114.252:45987
I20260925 22:54:49.418258 28163 heartbeater.cc:347] Connected to a master server at 127.20.114.253:35653
I20260925 22:54:49.418536 28161 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:49.418536 28163 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:49.419171 28161 heartbeater.cc:511] Master 127.20.114.252:45987 requested a full tablet report, sending...
I20260925 22:54:49.419186 28166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.400245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.20.114.194:51901)
0925 22:54:49.400706 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:49.400759 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:49.401929 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:49.402748 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:49.402751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:49.403037 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:49.403628 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.403645 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.405329 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.405332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.411868 (+ 6536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.411879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.412688 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.412692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.412791 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.413298 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:49.413314 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:49.414723 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:49.417360 (+ 2637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:49.417366 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:49.417371 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:49.417612 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:49.417981 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:49.417984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:49.417986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:49.418082 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:49.418643 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:49.418646 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:49.418729 (+ 83us) client_negotiation.cc:769] Sending connection context
0925 22:54:49.418854 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 22:54:49.419018 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":5851,"spinlock_wait_cycles":53376,"thread_start_us":138,"threads_started":1}
I20260925 22:54:49.419174 28163 heartbeater.cc:511] Master 127.20.114.253:35653 requested a full tablet report, sending...
I20260925 22:54:49.419543 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.399749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:51901 (local address 127.20.114.254:37819)
0925 22:54:49.399878 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:49.399883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:49.400857 (+ 974us) server_negotiation.cc:407] Connection header received
0925 22:54:49.402426 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:49.402429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:49.402483 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:49.402593 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:49.403812 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.404622 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.412038 (+ 7416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.412421 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.412514 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.414869 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:49.414872 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:49.414875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:49.414903 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:49.417246 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.417730 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.417734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.417735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.417828 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.418223 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.418228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.418229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.418500 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:49.418600 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:49.419291 (+ 691us) server_negotiation.cc:299] Negotiation successful
0925 22:54:49.419404 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:54:49.420394 27647 ts_manager.cc:194] Registered new tserver with Master: 92bfd5c259204d5994cfd0029b038c21 (127.20.114.194:43103)
I20260925 22:54:49.420394 27841 ts_manager.cc:194] Registered new tserver with Master: 92bfd5c259204d5994cfd0029b038c21 (127.20.114.194:43103)
I20260925 22:54:49.420589 28162 heartbeater.cc:347] Connected to a master server at 127.20.114.254:37819
I20260925 22:54:49.420668 28162 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:49.420842 28162 heartbeater.cc:511] Master 127.20.114.254:37819 requested a full tablet report, sending...
I20260925 22:54:49.421579 27054 ts_manager.cc:194] Re-registered known tserver with Master: 92bfd5c259204d5994cfd0029b038c21 (127.20.114.194:43103)
I20260925 22:54:49.422439 27054 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:51901
W20260925 22:54:49.527688 28168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:49.527935 28168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:49.527978 28168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:49.532172 28168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:49.532235 28168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:49.532294 28168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:54:49.537732 28168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.195.keytab
--principal=oryx/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:34885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=42699
--tserver_master_addrs=127.20.114.254:37819,127.20.114.253:35653,127.20.114.252:45987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:49.538751 28168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:49.539835 28168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:49.549531 28173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:49.549551 28174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:49.549719 28168 server_base.cc:1034] running on GCE node
W20260925 22:54:49.549718 28176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:49.550195 28168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:49.550784 28168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:49.551937 28168 hybrid_clock.cc:648] HybridClock initialized: now 1790376889551900 us; error 51 us; skew 500 ppm
Sep 25 22:54:49 dist-test-slave-x1lh krb5kdc[27009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376889, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:49.554554 28168 init.cc:377] Logged in from keytab as oryx/127.20.114.195@KRBTEST.COM (short username oryx)
I20260925 22:54:49.555369 28168 webserver.cc:460] Webserver started at http://127.20.114.195:42699/ using document root <none> and password file <none>
I20260925 22:54:49.555915 28168 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:49.555982 28168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:49.559888 28168 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:49.561872 28183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:49.563076 28168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:49.563189 28168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "a341f62c9efa4a52995a7670f9e61721"
format_stamp: "Formatted at 2026-09-25 22:54:48 on dist-test-slave-x1lh"
server_key: "71392de19e862dff0d89508e49ac7ec8"
server_key_iv: "ea52d2eb7db71b4e2515067af1a70147"
server_key_version: "encryptionkey@0"
I20260925 22:54:49.563705 28168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:49.586491 28168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:49.589653 28168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:49.589846 28168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:49.590461 28168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:49.591478 28168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:49.591530 28168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:49.591605 28168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:49.591638 28168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:49.600201 28168 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:34885
I20260925 22:54:49.600224 28296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:34885 every 8 connection(s)
I20260925 22:54:49.601362 28168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
Sep 25 22:54:49 dist-test-slave-x1lh krb5kdc[27009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376889, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.253@KRBTEST.COM
I20260925 22:54:49.611606 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 28168
Sep 25 22:54:49 dist-test-slave-x1lh krb5kdc[27009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376889, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM
Sep 25 22:54:49 dist-test-slave-x1lh krb5kdc[27009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376889, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.252@KRBTEST.COM
I20260925 22:54:49.618294 28302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.603654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:35653 (local address 127.20.114.195:43477)
0925 22:54:49.604640 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:49.604652 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:49.605526 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:49.606079 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:49.606086 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:49.606636 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:49.607172 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.607180 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.608254 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.608258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.609358 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.609368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.609970 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.609975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.610083 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.610525 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:49.610545 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:49.612133 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:49.614830 (+ 2697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:49.614835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:49.614853 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:49.615229 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:49.615658 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:49.615661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:49.615663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:49.615843 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:49.616322 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:49.616328 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:49.616654 (+ 326us) client_negotiation.cc:769] Sending connection context
0925 22:54:49.616895 (+ 241us) client_negotiation.cc:240] Negotiation successful
0925 22:54:49.617252 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":259,"spinlock_wait_cycles":38016,"thread_start_us":200,"threads_started":1}
I20260925 22:54:49.618722 28301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.603654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:45987 (local address 127.20.114.195:45109)
0925 22:54:49.604428 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:49.604478 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:49.605526 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:49.606079 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:49.606097 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:49.606636 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:49.607173 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.607180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.608254 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.608258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.609358 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.609368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.610085 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.610088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.610200 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.610679 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:49.610693 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:49.613024 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:49.616282 (+ 3258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:49.616288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:49.616291 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:49.616802 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:49.617106 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:49.617109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:49.617111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:49.617190 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:49.617581 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:49.617599 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:49.617699 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 22:54:49.618388 (+ 689us) client_negotiation.cc:240] Negotiation successful
0925 22:54:49.618512 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":161,"spinlock_wait_cycles":18048,"thread_start_us":237,"threads_started":1}
I20260925 22:54:49.619253 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.603291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:43477 (local address 127.20.114.253:35653)
0925 22:54:49.603422 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:49.603426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:49.604738 (+ 1312us) server_negotiation.cc:407] Connection header received
0925 22:54:49.605836 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:49.605840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:49.605891 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:49.605955 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:49.607315 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.608113 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.609512 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.609751 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.609807 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.612308 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:49.612312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:49.612315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:49.612343 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:49.614699 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.615358 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.615361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.615363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.615448 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.615957 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.615962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.615968 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.616205 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:49.618898 (+ 2693us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:49.618972 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 22:54:49.619104 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:54:49.619511 28028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.603173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:45109 (local address 127.20.114.252:45987)
0925 22:54:49.603309 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:49.603312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:49.604579 (+ 1267us) server_negotiation.cc:407] Connection header received
0925 22:54:49.605760 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:49.605763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:49.605821 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:49.605872 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:49.607316 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.608117 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.609522 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.609853 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.609959 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.613204 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:49.613210 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:49.613215 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:49.613246 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:49.616183 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.616924 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.616927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.616929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.617009 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.617281 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.617284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.617286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.617465 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:49.617540 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:49.619255 (+ 1715us) server_negotiation.cc:299] Negotiation successful
0925 22:54:49.619373 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:54:49.620230 28297 heartbeater.cc:347] Connected to a master server at 127.20.114.252:45987
I20260925 22:54:49.620489 28297 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:49.621061 28297 heartbeater.cc:511] Master 127.20.114.252:45987 requested a full tablet report, sending...
I20260925 22:54:49.620231 28299 heartbeater.cc:347] Connected to a master server at 127.20.114.253:35653
I20260925 22:54:49.621645 28299 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:49.621786 28299 heartbeater.cc:511] Master 127.20.114.253:35653 requested a full tablet report, sending...
I20260925 22:54:49.621942 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.605070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:51611 (local address 127.20.114.254:37819)
0925 22:54:49.605188 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:49.605192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:49.605214 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:54:49.605750 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:49.605754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:49.605810 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:49.605911 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:49.607300 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.607833 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.609029 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.609253 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.609366 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.613276 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:49.613282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:49.613287 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:49.613316 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:49.615175 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.620083 (+ 4908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.620087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.620088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.620180 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.620562 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.620848 (+ 286us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.620853 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.621102 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:49.621196 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:49.621650 (+ 454us) server_negotiation.cc:299] Negotiation successful
0925 22:54:49.621769 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:54:49.622542 27841 ts_manager.cc:194] Registered new tserver with Master: a341f62c9efa4a52995a7670f9e61721 (127.20.114.195:34885)
I20260925 22:54:49.622730 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.604791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.20.114.195:51611)
0925 22:54:49.605073 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:49.605086 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:49.605558 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:49.606077 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:49.606084 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:49.606615 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:49.607159 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.607172 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.607970 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.607973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.608849 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.608858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.609396 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.609400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.609637 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.610318 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:49.610342 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:49.613132 (+ 2790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:49.616308 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:49.616310 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:49.616312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:49.616555 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:49.620296 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:49.620354 (+ 58us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:49.620357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:49.620445 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:49.621322 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:49.621328 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:49.621433 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 22:54:49.622345 (+ 912us) client_negotiation.cc:240] Negotiation successful
0925 22:54:49.622498 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":55,"thread_start_us":108,"threads_started":1}
I20260925 22:54:49.622840 27647 ts_manager.cc:194] Registered new tserver with Master: a341f62c9efa4a52995a7670f9e61721 (127.20.114.195:34885)
I20260925 22:54:49.623499 28298 heartbeater.cc:347] Connected to a master server at 127.20.114.254:37819
I20260925 22:54:49.623569 28298 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:49.623760 28298 heartbeater.cc:511] Master 127.20.114.254:37819 requested a full tablet report, sending...
I20260925 22:54:49.624429 27054 ts_manager.cc:194] Re-registered known tserver with Master: a341f62c9efa4a52995a7670f9e61721 (127.20.114.195:34885)
I20260925 22:54:49.625211 27054 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:51611
I20260925 22:54:49.626848 28310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.614520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59176 (local address 127.20.114.254:37819)
0925 22:54:49.614787 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:49.614792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:49.615165 (+ 373us) server_negotiation.cc:407] Connection header received
0925 22:54:49.615343 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:49.615347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:49.615398 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:49.615459 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:49.616738 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.617572 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.618793 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.619173 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.619250 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.621899 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:49.621903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:49.621905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:49.621928 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:49.625074 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.625540 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.625546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.625550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.625635 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.625943 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.625946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.625948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.626149 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:49.626250 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:49.626589 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 22:54:49.626705 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
I20260925 22:54:49.628474 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.614124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58148 (local address 127.20.114.253:35653)
0925 22:54:49.619394 (+ 5270us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:49.619399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:49.619411 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:54:49.619445 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:49.619447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:49.619492 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:49.620396 (+ 904us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:49.620479 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.621187 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.622241 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.622505 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.622961 (+ 456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.624243 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:49.624248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:49.624254 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:49.624287 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:49.626111 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.626791 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.626798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.626802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.626892 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.627242 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.627244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.627246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.627450 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:49.628062 (+ 612us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:49.628138 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 22:54:49.628266 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5217,"thread_start_us":90,"threads_started":1}
I20260925 22:54:49.629406 28312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.615195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50374 (local address 127.20.114.252:45987)
0925 22:54:49.615735 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:49.615746 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:49.616106 (+ 360us) server_negotiation.cc:407] Connection header received
0925 22:54:49.616141 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:49.616144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:49.616188 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:49.616268 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:49.617168 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.617910 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.619114 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.619449 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.619559 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.624900 (+ 5341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:49.624907 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:49.624913 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:49.624940 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:49.627665 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.628122 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.628125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.628127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.628213 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:49.628492 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:49.628496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:49.628498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:49.628725 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:49.628842 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:49.629115 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 22:54:49.629244 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":84,"threads_started":1}
I20260925 22:54:49.632645 28028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.626253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50376 (local address 127.20.114.252:45987)
0925 22:54:49.626371 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:49.626378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:49.628798 (+ 2420us) server_negotiation.cc:407] Connection header received
0925 22:54:49.629007 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:49.629013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:49.629078 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:49.629163 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:49.629650 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.630255 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.631788 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.632188 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.632280 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.632380 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:49.632506 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 22:54:49.632562 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:54:49.632689 28314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.625936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:45987 (local address 127.0.0.1:50376)
0925 22:54:49.628723 (+ 2787us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:49.628738 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:49.628916 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:49.629268 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:49.629274 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:49.629389 (+ 115us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:49.629507 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.629517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.630564 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.630567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.631667 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.631677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.632300 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.632303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.632391 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.632413 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:54:49.632489 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:54:49.632554 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2720,"thread_start_us":100,"threads_started":1}
I20260925 22:54:49.633157 27855 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 22:54:49.633409 27887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c96778fffbfd40f186940fb51039e3ae. Latest consensus state: current_term: 1 leader_uuid: "c96778fffbfd40f186940fb51039e3ae" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: NON_VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: true } } }
I20260925 22:54:49.633556 27887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [sys.catalog]: This master's current role is: LEARNER
I20260925 22:54:49.630908 27054 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:54:49.634701 27054 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:54:49.637207 27809 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c96778fffbfd40f186940fb51039e3ae" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: NON_VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: true } } }
I20260925 22:54:49.637346 27809 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:49.638764 27809 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae: attempting to promote NON_VOTER 1fc6c03407644ff6b318682cd66988a9 to VOTER
I20260925 22:54:49.639087 27809 consensus_queue.cc:237] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: false } }
I20260925 22:54:49.640611 27855 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [term 1 LEARNER]: Refusing update from remote peer c96778fffbfd40f186940fb51039e3ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 22:54:49.641049 28316 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 22:54:49.642668 27809 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1fc6c03407644ff6b318682cd66988a9 (127.20.114.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: false } } }
I20260925 22:54:49.643044 27855 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1fc6c03407644ff6b318682cd66988a9 (127.20.114.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: false } } }
I20260925 22:54:49.644066 27809 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c96778fffbfd40f186940fb51039e3ae" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: false } } }
I20260925 22:54:49.644160 27809 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:49.644261 27809 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c96778fffbfd40f186940fb51039e3ae" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: false } } }
I20260925 22:54:49.644325 27809 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:49.644541 27887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c96778fffbfd40f186940fb51039e3ae" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: false } } }
I20260925 22:54:49.644649 27887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:54:49.647572 27664 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [term 1 FOLLOWER]: Refusing update from remote peer c96778fffbfd40f186940fb51039e3ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 22:54:49.649045 28318 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 22:54:49.650695 27664 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1fc6c03407644ff6b318682cd66988a9 (127.20.114.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: false } } }
I20260925 22:54:49.652966 27809 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c96778fffbfd40f186940fb51039e3ae" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: false } } }
I20260925 22:54:49.653121 27809 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:49.654484 28326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c96778fffbfd40f186940fb51039e3ae" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: false } } }
I20260925 22:54:49.654692 28326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:54:49.659047 28314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.654583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:33393 (local address 127.0.0.1:46432)
0925 22:54:49.654748 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:49.654766 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:49.654849 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:49.655549 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:49.655552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:49.655566 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:49.655672 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.655679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.656971 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.656975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.657720 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.657728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.658646 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.658649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.658743 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.658767 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:54:49.658847 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 22:54:49.658919 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 22:54:49.659109 28327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.654729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.20.114.193:33393)
0925 22:54:49.655008 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:49.655020 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:49.655052 (+ 32us) server_negotiation.cc:407] Connection header received
0925 22:54:49.655126 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:49.655131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:49.655383 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:49.655585 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:49.655764 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.656830 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.657839 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.658518 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.658637 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.658768 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:49.658874 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 22:54:49.658958 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":98,"threads_started":1}
I20260925 22:54:49.661274 28328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.655064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:34885 (local address 127.0.0.1:57894)
0925 22:54:49.655315 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:49.655329 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:49.655410 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:49.656831 (+ 1421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:49.656835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:49.656852 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:49.656980 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.656987 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.658302 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.658305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.659662 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.659671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.660900 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.660904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.660995 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.661016 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:54:49.661087 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 22:54:49.661152 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":107,"threads_started":1}
I20260925 22:54:49.661722 28330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.655940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57894 (local address 127.20.114.195:34885)
0925 22:54:49.656209 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:49.656221 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:49.656243 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:54:49.656315 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:49.656319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:49.656516 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:49.656684 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:49.657113 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.658187 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.659793 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.660589 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.660704 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.660814 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:49.661483 (+ 669us) server_negotiation.cc:299] Negotiation successful
0925 22:54:49.661590 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":103,"threads_started":1}
I20260925 22:54:49.662015 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.655427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:43103 (local address 127.0.0.1:59448)
0925 22:54:49.657301 (+ 1874us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:49.657314 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:49.657391 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:49.658148 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:49.658150 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:49.658167 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:49.658262 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.658269 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.659765 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.659768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.660554 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.660563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.661623 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.661625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.661702 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.661719 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 22:54:49.661810 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 22:54:49.661888 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1821,"thread_start_us":83,"threads_started":1}
I20260925 22:54:49.662420 28331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.656301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59448 (local address 127.20.114.194:43103)
0925 22:54:49.657617 (+ 1316us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:49.657632 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:49.657652 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:54:49.657718 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:49.657723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:49.657891 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:49.658020 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:49.658354 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.659643 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.660689 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.661306 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.661451 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.661561 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:49.662045 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 22:54:49.662293 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1223,"thread_start_us":81,"threads_started":1}
I20260925 22:54:49.662382 27958 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd5cfea046b432697403e6ffb59d82e (DEFAULT_TABLE table=test-table [id=cf8eba0aeb8144899445193f79d85608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:49.663527 27958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bd5cfea046b432697403e6ffb59d82e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:49.664320 28231 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd5cfea046b432697403e6ffb59d82e (DEFAULT_TABLE table=test-table [id=cf8eba0aeb8144899445193f79d85608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:49.665263 28231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bd5cfea046b432697403e6ffb59d82e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:49.667472 28095 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd5cfea046b432697403e6ffb59d82e (DEFAULT_TABLE table=test-table [id=cf8eba0aeb8144899445193f79d85608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:49.668391 28095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bd5cfea046b432697403e6ffb59d82e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:49.671320 28332 tablet_bootstrap.cc:492] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f: Bootstrap starting.
I20260925 22:54:49.672456 28333 tablet_bootstrap.cc:492] T 5bd5cfea046b432697403e6ffb59d82e P a341f62c9efa4a52995a7670f9e61721: Bootstrap starting.
I20260925 22:54:49.674005 28332 tablet_bootstrap.cc:654] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:49.674453 28333 tablet_bootstrap.cc:654] T 5bd5cfea046b432697403e6ffb59d82e P a341f62c9efa4a52995a7670f9e61721: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:49.674878 28332 log.cc:824] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:49.675225 28333 log.cc:824] T 5bd5cfea046b432697403e6ffb59d82e P a341f62c9efa4a52995a7670f9e61721: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:49.675858 28334 tablet_bootstrap.cc:492] T 5bd5cfea046b432697403e6ffb59d82e P 92bfd5c259204d5994cfd0029b038c21: Bootstrap starting.
I20260925 22:54:49.676993 28333 tablet_bootstrap.cc:492] T 5bd5cfea046b432697403e6ffb59d82e P a341f62c9efa4a52995a7670f9e61721: No bootstrap required, opened a new log
I20260925 22:54:49.677201 28333 ts_tablet_manager.cc:1397] T 5bd5cfea046b432697403e6ffb59d82e P a341f62c9efa4a52995a7670f9e61721: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:54:49.677574 28334 tablet_bootstrap.cc:654] T 5bd5cfea046b432697403e6ffb59d82e P 92bfd5c259204d5994cfd0029b038c21: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:49.678297 28334 log.cc:824] T 5bd5cfea046b432697403e6ffb59d82e P 92bfd5c259204d5994cfd0029b038c21: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:49.678457 28332 tablet_bootstrap.cc:492] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f: No bootstrap required, opened a new log
I20260925 22:54:49.678649 28332 ts_tablet_manager.cc:1397] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260925 22:54:49.680150 28334 tablet_bootstrap.cc:492] T 5bd5cfea046b432697403e6ffb59d82e P 92bfd5c259204d5994cfd0029b038c21: No bootstrap required, opened a new log
I20260925 22:54:49.680359 28334 ts_tablet_manager.cc:1397] T 5bd5cfea046b432697403e6ffb59d82e P 92bfd5c259204d5994cfd0029b038c21: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:54:49.680663 28333 raft_consensus.cc:348] T 5bd5cfea046b432697403e6ffb59d82e P a341f62c9efa4a52995a7670f9e61721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a341f62c9efa4a52995a7670f9e61721" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 34885 } } peers { permanent_uuid: "92bfd5c259204d5994cfd0029b038c21" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43103 } } peers { permanent_uuid: "c01ee4ba045949eebcc2660e769f7b6f" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 33393 } }
I20260925 22:54:49.681020 28333 raft_consensus.cc:374] T 5bd5cfea046b432697403e6ffb59d82e P a341f62c9efa4a52995a7670f9e61721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:49.681095 28333 raft_consensus.cc:729] T 5bd5cfea046b432697403e6ffb59d82e P a341f62c9efa4a52995a7670f9e61721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a341f62c9efa4a52995a7670f9e61721, State: Initialized, Role: FOLLOWER
I20260925 22:54:49.681529 28333 consensus_queue.cc:260] T 5bd5cfea046b432697403e6ffb59d82e P a341f62c9efa4a52995a7670f9e61721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a341f62c9efa4a52995a7670f9e61721" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 34885 } } peers { permanent_uuid: "92bfd5c259204d5994cfd0029b038c21" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43103 } } peers { permanent_uuid: "c01ee4ba045949eebcc2660e769f7b6f" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 33393 } }
I20260925 22:54:49.682516 28333 ts_tablet_manager.cc:1428] T 5bd5cfea046b432697403e6ffb59d82e P a341f62c9efa4a52995a7670f9e61721: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:54:49.682906 28298 heartbeater.cc:503] Master 127.20.114.254:37819 was elected leader, sending a full tablet report...
I20260925 22:54:49.684835 28332 raft_consensus.cc:348] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a341f62c9efa4a52995a7670f9e61721" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 34885 } } peers { permanent_uuid: "92bfd5c259204d5994cfd0029b038c21" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43103 } } peers { permanent_uuid: "c01ee4ba045949eebcc2660e769f7b6f" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 33393 } }
I20260925 22:54:49.685109 28332 raft_consensus.cc:374] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:49.685159 28332 raft_consensus.cc:729] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c01ee4ba045949eebcc2660e769f7b6f, State: Initialized, Role: FOLLOWER
I20260925 22:54:49.685236 28334 raft_consensus.cc:348] T 5bd5cfea046b432697403e6ffb59d82e P 92bfd5c259204d5994cfd0029b038c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a341f62c9efa4a52995a7670f9e61721" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 34885 } } peers { permanent_uuid: "92bfd5c259204d5994cfd0029b038c21" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43103 } } peers { permanent_uuid: "c01ee4ba045949eebcc2660e769f7b6f" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 33393 } }
I20260925 22:54:49.685438 28334 raft_consensus.cc:374] T 5bd5cfea046b432697403e6ffb59d82e P 92bfd5c259204d5994cfd0029b038c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:49.685482 28334 raft_consensus.cc:729] T 5bd5cfea046b432697403e6ffb59d82e P 92bfd5c259204d5994cfd0029b038c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92bfd5c259204d5994cfd0029b038c21, State: Initialized, Role: FOLLOWER
I20260925 22:54:49.685642 28332 consensus_queue.cc:260] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a341f62c9efa4a52995a7670f9e61721" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 34885 } } peers { permanent_uuid: "92bfd5c259204d5994cfd0029b038c21" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43103 } } peers { permanent_uuid: "c01ee4ba045949eebcc2660e769f7b6f" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 33393 } }
I20260925 22:54:49.685875 28334 consensus_queue.cc:260] T 5bd5cfea046b432697403e6ffb59d82e P 92bfd5c259204d5994cfd0029b038c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a341f62c9efa4a52995a7670f9e61721" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 34885 } } peers { permanent_uuid: "92bfd5c259204d5994cfd0029b038c21" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43103 } } peers { permanent_uuid: "c01ee4ba045949eebcc2660e769f7b6f" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 33393 } }
I20260925 22:54:49.686578 28332 ts_tablet_manager.cc:1428] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260925 22:54:49.686663 28334 ts_tablet_manager.cc:1428] T 5bd5cfea046b432697403e6ffb59d82e P 92bfd5c259204d5994cfd0029b038c21: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:54:49.686957 28162 heartbeater.cc:503] Master 127.20.114.254:37819 was elected leader, sending a full tablet report...
I20260925 22:54:49.687146 28025 heartbeater.cc:503] Master 127.20.114.254:37819 was elected leader, sending a full tablet report...
W20260925 22:54:49.698755 28027 tablet.cc:2369] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:54:49.854784 28300 tablet.cc:2369] T 5bd5cfea046b432697403e6ffb59d82e P a341f62c9efa4a52995a7670f9e61721: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:54:49.899782 28164 tablet.cc:2369] T 5bd5cfea046b432697403e6ffb59d82e P 92bfd5c259204d5994cfd0029b038c21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:54:49.925164 28339 raft_consensus.cc:482] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:54:49.925410 28339 raft_consensus.cc:504] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a341f62c9efa4a52995a7670f9e61721" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 34885 } } peers { permanent_uuid: "92bfd5c259204d5994cfd0029b038c21" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43103 } } peers { permanent_uuid: "c01ee4ba045949eebcc2660e769f7b6f" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 33393 } }
I20260925 22:54:49.926566 28339 leader_election.cc:288] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a341f62c9efa4a52995a7670f9e61721 (127.20.114.195:34885), 92bfd5c259204d5994cfd0029b038c21 (127.20.114.194:43103)
I20260925 22:54:49.930672 28331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.927391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:47633 (local address 127.20.114.194:43103)
0925 22:54:49.927511 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:49.927515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:49.927645 (+ 130us) server_negotiation.cc:407] Connection header received
0925 22:54:49.927736 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:49.927739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:49.927790 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:49.927869 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:49.928143 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.928649 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.929740 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.930114 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.930225 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.930278 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:49.930452 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 22:54:49.930525 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:54:49.930696 28330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.927009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:48635 (local address 127.20.114.195:34885)
0925 22:54:49.927157 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:49.927162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:49.927287 (+ 125us) server_negotiation.cc:407] Connection header received
0925 22:54:49.927455 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:49.927459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:49.927510 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:49.927616 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:49.927925 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.928512 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.929737 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.930134 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.930265 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.930317 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:49.930515 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 22:54:49.930594 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:54:49.930686 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.927300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:43103 (local address 127.20.114.193:47633)
0925 22:54:49.927563 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:49.927577 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:49.927657 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:49.927915 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:49.927918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:49.927934 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:49.928029 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.928035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.928761 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.928764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.929621 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.929630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.930229 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.930232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.930322 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.930350 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 22:54:49.930417 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 22:54:49.930494 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":28,"thread_start_us":61,"threads_started":1}
I20260925 22:54:49.930698 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:49.926923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:34885 (local address 127.20.114.193:48635)
0925 22:54:49.927195 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:49.927208 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:49.927349 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:49.927660 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:49.927667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:49.927691 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:49.927817 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.927827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.928611 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.928615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.929619 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:49.929627 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:49.930274 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:49.930277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:49.930377 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:49.930393 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 22:54:49.930488 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 22:54:49.930551 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":25,"thread_start_us":89,"threads_started":1}
I20260925 22:54:49.931260 28115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd5cfea046b432697403e6ffb59d82e" candidate_uuid: "c01ee4ba045949eebcc2660e769f7b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92bfd5c259204d5994cfd0029b038c21" is_pre_election: true
I20260925 22:54:49.931464 28251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd5cfea046b432697403e6ffb59d82e" candidate_uuid: "c01ee4ba045949eebcc2660e769f7b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a341f62c9efa4a52995a7670f9e61721" is_pre_election: true
I20260925 22:54:49.931560 28115 raft_consensus.cc:2454] T 5bd5cfea046b432697403e6ffb59d82e P 92bfd5c259204d5994cfd0029b038c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c01ee4ba045949eebcc2660e769f7b6f in term 0.
I20260925 22:54:49.931761 28251 raft_consensus.cc:2454] T 5bd5cfea046b432697403e6ffb59d82e P a341f62c9efa4a52995a7670f9e61721 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c01ee4ba045949eebcc2660e769f7b6f in term 0.
I20260925 22:54:49.932065 27913 leader_election.cc:302] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92bfd5c259204d5994cfd0029b038c21, c01ee4ba045949eebcc2660e769f7b6f; no voters:
I20260925 22:54:49.932318 28339 raft_consensus.cc:2781] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:54:49.932412 28339 raft_consensus.cc:482] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:54:49.932468 28339 raft_consensus.cc:3037] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:49.933533 28339 raft_consensus.cc:504] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a341f62c9efa4a52995a7670f9e61721" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 34885 } } peers { permanent_uuid: "92bfd5c259204d5994cfd0029b038c21" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43103 } } peers { permanent_uuid: "c01ee4ba045949eebcc2660e769f7b6f" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 33393 } }
I20260925 22:54:49.933897 28339 leader_election.cc:288] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f [CANDIDATE]: Term 1 election: Requested vote from peers a341f62c9efa4a52995a7670f9e61721 (127.20.114.195:34885), 92bfd5c259204d5994cfd0029b038c21 (127.20.114.194:43103)
I20260925 22:54:49.934176 28251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd5cfea046b432697403e6ffb59d82e" candidate_uuid: "c01ee4ba045949eebcc2660e769f7b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a341f62c9efa4a52995a7670f9e61721"
I20260925 22:54:49.934223 28115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd5cfea046b432697403e6ffb59d82e" candidate_uuid: "c01ee4ba045949eebcc2660e769f7b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92bfd5c259204d5994cfd0029b038c21"
I20260925 22:54:49.934302 28251 raft_consensus.cc:3037] T 5bd5cfea046b432697403e6ffb59d82e P a341f62c9efa4a52995a7670f9e61721 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:49.934320 28115 raft_consensus.cc:3037] T 5bd5cfea046b432697403e6ffb59d82e P 92bfd5c259204d5994cfd0029b038c21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:49.935509 28251 raft_consensus.cc:2454] T 5bd5cfea046b432697403e6ffb59d82e P a341f62c9efa4a52995a7670f9e61721 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c01ee4ba045949eebcc2660e769f7b6f in term 1.
I20260925 22:54:49.935508 28115 raft_consensus.cc:2454] T 5bd5cfea046b432697403e6ffb59d82e P 92bfd5c259204d5994cfd0029b038c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c01ee4ba045949eebcc2660e769f7b6f in term 1.
I20260925 22:54:49.935797 27913 leader_election.cc:302] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92bfd5c259204d5994cfd0029b038c21, c01ee4ba045949eebcc2660e769f7b6f; no voters:
I20260925 22:54:49.935969 28339 raft_consensus.cc:2781] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:49.936144 28339 raft_consensus.cc:686] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f [term 1 LEADER]: Becoming Leader. State: Replica: c01ee4ba045949eebcc2660e769f7b6f, State: Running, Role: LEADER
I20260925 22:54:49.936514 28339 consensus_queue.cc:237] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a341f62c9efa4a52995a7670f9e61721" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 34885 } } peers { permanent_uuid: "92bfd5c259204d5994cfd0029b038c21" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43103 } } peers { permanent_uuid: "c01ee4ba045949eebcc2660e769f7b6f" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 33393 } }
I20260925 22:54:49.939949 27053 catalog_manager.cc:5495] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f reported cstate change: term changed from 0 to 1, leader changed from <none> to c01ee4ba045949eebcc2660e769f7b6f (127.20.114.193). New cstate: current_term: 1 leader_uuid: "c01ee4ba045949eebcc2660e769f7b6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a341f62c9efa4a52995a7670f9e61721" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 34885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92bfd5c259204d5994cfd0029b038c21" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c01ee4ba045949eebcc2660e769f7b6f" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 33393 } health_report { overall_health: HEALTHY } } }
I20260925 22:54:50.020869 28327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:50.017603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46442 (local address 127.20.114.193:33393)
0925 22:54:50.017728 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:50.017732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:50.017746 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:54:50.017818 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:50.017821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:50.017872 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:50.017961 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:50.018220 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:50.018719 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:50.019515 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:50.019724 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:50.019797 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:50.020073 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:50.020480 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:50.020583 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:50.020651 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 22:54:50.020699 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:54:50.026647 28251 raft_consensus.cc:1261] T 5bd5cfea046b432697403e6ffb59d82e P a341f62c9efa4a52995a7670f9e61721 [term 1 FOLLOWER]: Refusing update from remote peer c01ee4ba045949eebcc2660e769f7b6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:50.026647 28115 raft_consensus.cc:1261] T 5bd5cfea046b432697403e6ffb59d82e P 92bfd5c259204d5994cfd0029b038c21 [term 1 FOLLOWER]: Refusing update from remote peer c01ee4ba045949eebcc2660e769f7b6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:50.027403 28343 consensus_queue.cc:1035] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "92bfd5c259204d5994cfd0029b038c21" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:50.027638 28339 consensus_queue.cc:1035] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a341f62c9efa4a52995a7670f9e61721" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 34885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:50.038998 28347 mvcc.cc:205] Tried to move back new op lower bound from 7333383741541249024 to 7333383741182603264. Current Snapshot: MvccSnapshot[applied={T|T < 7333383741541249024}]
I20260925 22:54:50.047652 27053 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:50.047809 27053 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:50.051981 27053 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c96778fffbfd40f186940fb51039e3ae: Sending DeleteTablet for 3 replicas of tablet 5bd5cfea046b432697403e6ffb59d82e
I20260925 22:54:50.053126 27958 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd5cfea046b432697403e6ffb59d82e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:50 UTC) from {username='oryx', principal='oryx/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:46432
I20260925 22:54:50.054350 28231 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd5cfea046b432697403e6ffb59d82e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:50 UTC) from {username='oryx', principal='oryx/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:57894
I20260925 22:54:50.054807 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 27022
I20260925 22:54:50.054840 28095 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd5cfea046b432697403e6ffb59d82e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:50 UTC) from {username='oryx', principal='oryx/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:59448
I20260925 22:54:50.054854 28358 tablet_replica.cc:330] T 5bd5cfea046b432697403e6ffb59d82e P a341f62c9efa4a52995a7670f9e61721: stopping tablet replica
I20260925 22:54:50.056260 28358 raft_consensus.cc:2229] T 5bd5cfea046b432697403e6ffb59d82e P a341f62c9efa4a52995a7670f9e61721 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:50.056668 28358 raft_consensus.cc:2258] T 5bd5cfea046b432697403e6ffb59d82e P a341f62c9efa4a52995a7670f9e61721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:50.058173 28358 ts_tablet_manager.cc:1905] T 5bd5cfea046b432697403e6ffb59d82e P a341f62c9efa4a52995a7670f9e61721: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:50.060906 28358 ts_tablet_manager.cc:1918] T 5bd5cfea046b432697403e6ffb59d82e P a341f62c9efa4a52995a7670f9e61721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:54:50.061065 28358 log.cc:1196] T 5bd5cfea046b432697403e6ffb59d82e P a341f62c9efa4a52995a7670f9e61721: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/5bd5cfea046b432697403e6ffb59d82e
I20260925 22:54:50.061481 28358 ts_tablet_manager.cc:1939] T 5bd5cfea046b432697403e6ffb59d82e P a341f62c9efa4a52995a7670f9e61721: Deleting consensus metadata
W20260925 22:54:50.063056 28186 connection.cc:535] server connection from 127.0.0.1:57894 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 22:54:50.063469 27639 connection.cc:535] server connection from 127.0.0.1:58134 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 22:54:50.064927 28359 tablet_replica.cc:330] T 5bd5cfea046b432697403e6ffb59d82e P 92bfd5c259204d5994cfd0029b038c21: stopping tablet replica
I20260925 22:54:50.065279 28359 raft_consensus.cc:2229] T 5bd5cfea046b432697403e6ffb59d82e P 92bfd5c259204d5994cfd0029b038c21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:50.065665 28359 raft_consensus.cc:2258] T 5bd5cfea046b432697403e6ffb59d82e P 92bfd5c259204d5994cfd0029b038c21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:50.066051 28356 tablet_replica.cc:330] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f: stopping tablet replica
I20260925 22:54:50.066326 28356 raft_consensus.cc:2229] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:54:50.066787 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:50.054357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58158 (local address 127.20.114.253:35653)
0925 22:54:50.054504 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:50.054509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:50.056184 (+ 1675us) server_negotiation.cc:407] Connection header received
0925 22:54:50.056379 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:50.056385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:50.056443 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:50.056495 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:50.057795 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:50.058704 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:50.060120 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:50.060543 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:50.060636 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:50.062787 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:50.062794 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:50.062800 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:50.062834 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:50.064658 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:50.065259 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:50.065263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:50.065265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:50.065349 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:50.065725 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:50.065730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:50.065734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:50.065940 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:50.066046 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:50.066504 (+ 458us) server_negotiation.cc:299] Negotiation successful
0925 22:54:50.066629 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:54:50.069538 28359 ts_tablet_manager.cc:1905] T 5bd5cfea046b432697403e6ffb59d82e P 92bfd5c259204d5994cfd0029b038c21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:50.066768 28356 raft_consensus.cc:2258] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:50.073572 28356 ts_tablet_manager.cc:1905] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:50.076260 28356 ts_tablet_manager.cc:1918] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:54:50.076352 28356 log.cc:1196] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-0/wal/wals/5bd5cfea046b432697403e6ffb59d82e
I20260925 22:54:50.076709 28356 ts_tablet_manager.cc:1939] T 5bd5cfea046b432697403e6ffb59d82e P c01ee4ba045949eebcc2660e769f7b6f: Deleting consensus metadata
I20260925 22:54:50.077384 28028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:50.054471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50384 (local address 127.20.114.252:45987)
0925 22:54:50.054572 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:50.054576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:50.067513 (+ 12937us) server_negotiation.cc:407] Connection header received
0925 22:54:50.067684 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:50.067687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:50.067735 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:50.067785 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:50.068648 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:50.069501 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:50.071144 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:50.071501 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:50.071572 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:50.073436 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:50.073440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:50.073443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:50.073468 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:50.075245 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:50.075710 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:50.075713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:50.075715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:50.075796 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:50.076068 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:50.076071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:50.076073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:50.076277 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:50.076377 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:50.076784 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 22:54:50.076922 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 22:54:50.077849 27913 connection.cc:441] server connection from 127.0.0.1:46432 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46432 (request call id 1) could send its response
I20260925 22:54:50.079151 28359 ts_tablet_manager.cc:1918] T 5bd5cfea046b432697403e6ffb59d82e P 92bfd5c259204d5994cfd0029b038c21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:54:50.079268 28359 log.cc:1196] T 5bd5cfea046b432697403e6ffb59d82e P 92bfd5c259204d5994cfd0029b038c21: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/5bd5cfea046b432697403e6ffb59d82e
I20260925 22:54:50.079636 28359 ts_tablet_manager.cc:1939] T 5bd5cfea046b432697403e6ffb59d82e P 92bfd5c259204d5994cfd0029b038c21: Deleting consensus metadata
W20260925 22:54:50.080837 28050 connection.cc:441] server connection from 127.0.0.1:59448 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59448 (request call id 1) could send its response
I20260925 22:54:50.082903 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:50.068050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58170 (local address 127.20.114.253:35653)
0925 22:54:50.068157 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:50.068161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:50.069206 (+ 1045us) server_negotiation.cc:407] Connection header received
0925 22:54:50.069741 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:50.069744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:50.069785 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:50.069829 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:50.071909 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:50.072625 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:50.076176 (+ 3551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:50.076506 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:50.076597 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:50.078901 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:50.078907 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:50.078909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:50.078940 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:50.081035 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:50.081568 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:50.081572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:50.081574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:50.081664 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:50.081971 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:50.081975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:50.081977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:50.082187 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:50.082349 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:50.082598 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 22:54:50.082712 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 22:54:50.083793 28312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:50.067484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50388 (local address 127.20.114.252:45987)
0925 22:54:50.067734 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:50.067738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:50.068996 (+ 1258us) server_negotiation.cc:407] Connection header received
0925 22:54:50.069467 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:50.069470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:50.069516 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:50.069570 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:50.074262 (+ 4692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:50.074983 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:50.077144 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:50.077459 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:50.077531 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:50.080192 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:50.080196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:50.080198 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:50.080221 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:50.081878 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:50.082316 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:50.082320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:50.082322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:50.082420 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:50.082706 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:50.082725 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:50.082728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:50.082922 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:50.083026 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:50.083457 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 22:54:50.083623 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260925 22:54:51.039990 28367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.039416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.20.114.194:46153)
0925 22:54:51.039760 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.039859 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":75,"threads_started":1}
W20260925 22:54:51.040674 28162 heartbeater.cc:644] Failed to heartbeat to 127.20.114.254:37819 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
I20260925 22:54:51.041405 28368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.040944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.20.114.195:51823)
0925 22:54:51.041196 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.041280 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":58,"threads_started":1}
W20260925 22:54:51.041700 28298 heartbeater.cc:644] Failed to heartbeat to 127.20.114.254:37819 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
I20260925 22:54:51.041951 28369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.041596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.20.114.193:32825)
0925 22:54:51.041799 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.041880 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
W20260925 22:54:51.042188 28025 heartbeater.cc:644] Failed to heartbeat to 127.20.114.254:37819 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
I20260925 22:54:51.552740 28370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c96778fffbfd40f186940fb51039e3ae)
I20260925 22:54:51.552992 28370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: false } }
I20260925 22:54:51.553987 28370 leader_election.cc:288] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c96778fffbfd40f186940fb51039e3ae (127.20.114.254:37819), 88d59d46332e40e1af313320f228c2dc (127.20.114.253:35653)
I20260925 22:54:51.554558 28371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.554152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.0.0.1:59642)
0925 22:54:51.554363 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.554455 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
W20260925 22:54:51.554762 27832 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
I20260925 22:54:51.555573 28371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.555372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.0.0.1:59656)
0925 22:54:51.555479 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.555522 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 22:54:51.555719 27832 leader_election.cc:334] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c96778fffbfd40f186940fb51039e3ae (127.20.114.254:37819): Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
I20260925 22:54:51.556928 28375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c96778fffbfd40f186940fb51039e3ae)
I20260925 22:54:51.557097 28375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: false } }
I20260925 22:54:51.558032 28375 leader_election.cc:288] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c96778fffbfd40f186940fb51039e3ae (127.20.114.254:37819), 1fc6c03407644ff6b318682cd66988a9 (127.20.114.252:45987)
I20260925 22:54:51.558454 28376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.558112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.0.0.1:59662)
0925 22:54:51.558295 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.558363 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":54,"threads_started":1}
I20260925 22:54:51.558712 28373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.554290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58174 (local address 127.20.114.253:35653)
0925 22:54:51.554586 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:51.554591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:51.554622 (+ 31us) server_negotiation.cc:407] Connection header received
0925 22:54:51.555164 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:51.555167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:51.555224 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:51.555321 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:51.555664 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.556313 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.557693 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.558137 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.558275 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:51.558327 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:51.558483 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 22:54:51.558593 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":90,"threads_started":1}
I20260925 22:54:51.558921 28372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.554228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:35653 (local address 127.0.0.1:58174)
0925 22:54:51.554456 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.554472 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:51.554577 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:51.555349 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:51.555353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:51.555374 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:51.555492 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:51.555497 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.556414 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.556417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:51.557555 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:51.557571 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.558263 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.558267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:51.558355 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:51.558376 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:54:51.558469 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 22:54:51.558650 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":53,"threads_started":1}
W20260925 22:54:51.558939 27642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
I20260925 22:54:51.559369 27664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1fc6c03407644ff6b318682cd66988a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "88d59d46332e40e1af313320f228c2dc" is_pre_election: true
I20260925 22:54:51.559571 27664 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fc6c03407644ff6b318682cd66988a9 in term 1.
I20260925 22:54:51.559850 28377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.559620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.0.0.1:59674)
0925 22:54:51.559733 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.559785 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 22:54:51.559942 27642 leader_election.cc:334] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c96778fffbfd40f186940fb51039e3ae (127.20.114.254:37819): Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
I20260925 22:54:51.559926 27831 leader_election.cc:302] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fc6c03407644ff6b318682cd66988a9, 88d59d46332e40e1af313320f228c2dc; no voters: c96778fffbfd40f186940fb51039e3ae
I20260925 22:54:51.560150 28370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 22:54:51.560213 28370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c96778fffbfd40f186940fb51039e3ae)
I20260925 22:54:51.560271 28370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:54:51.561553 28370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: false } }
I20260925 22:54:51.561966 28370 leader_election.cc:288] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [CANDIDATE]: Term 2 election: Requested vote from peers c96778fffbfd40f186940fb51039e3ae (127.20.114.254:37819), 88d59d46332e40e1af313320f228c2dc (127.20.114.253:35653)
I20260925 22:54:51.562281 28372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.562043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.0.0.1:59684)
0925 22:54:51.562167 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.562217 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 22:54:51.562269 27664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1fc6c03407644ff6b318682cd66988a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "88d59d46332e40e1af313320f228c2dc"
I20260925 22:54:51.562373 27664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:54:51.562618 28376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.558383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:45987 (local address 127.0.0.1:50400)
0925 22:54:51.558633 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.558647 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:51.558728 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:51.559132 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:51.559138 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:51.559163 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:51.559250 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:51.559258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.560294 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.560300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:51.561495 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:51.561506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.562273 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.562277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:51.562382 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:51.562406 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:54:51.562480 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 22:54:51.562527 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":46,"threads_started":1}
I20260925 22:54:51.562732 28378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.558698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50400 (local address 127.20.114.252:45987)
0925 22:54:51.558911 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:51.558920 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:51.558931 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:54:51.558970 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:51.558972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:51.559013 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:51.559358 (+ 345us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:51.559448 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.560144 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.561616 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.562169 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.562288 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:51.562338 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:51.562516 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 22:54:51.562622 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":90,"threads_started":1}
I20260925 22:54:51.563387 27664 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fc6c03407644ff6b318682cd66988a9 in term 2.
I20260925 22:54:51.563644 27831 leader_election.cc:302] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fc6c03407644ff6b318682cd66988a9, 88d59d46332e40e1af313320f228c2dc; no voters:
I20260925 22:54:51.563813 28370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:54:51.563891 27855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88d59d46332e40e1af313320f228c2dc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1fc6c03407644ff6b318682cd66988a9" is_pre_election: true
I20260925 22:54:51.563992 28370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [term 2 LEADER]: Becoming Leader. State: Replica: 1fc6c03407644ff6b318682cd66988a9, State: Running, Role: LEADER
I20260925 22:54:51.564109 28372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.563881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.0.0.1:59696)
0925 22:54:51.563981 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.564038 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 22:54:51.564254 28370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: false } }
W20260925 22:54:51.564319 27832 leader_election.cc:334] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c96778fffbfd40f186940fb51039e3ae (127.20.114.254:37819): Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
I20260925 22:54:51.565341 28380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fc6c03407644ff6b318682cd66988a9. Latest consensus state: current_term: 2 leader_uuid: "1fc6c03407644ff6b318682cd66988a9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: false } } }
I20260925 22:54:51.565481 28380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:51.565814 28382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:54:51.565958 27642 leader_election.cc:302] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88d59d46332e40e1af313320f228c2dc; no voters: 1fc6c03407644ff6b318682cd66988a9, c96778fffbfd40f186940fb51039e3ae
I20260925 22:54:51.566169 28375 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:54:51.568091 28382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:54:51.568472 28382 catalog_manager.cc:1259] Loaded cluster ID: 0caef3ffb432457ba13ec0443bf9ca8a
I20260925 22:54:51.568518 28382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:51.569087 28382 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:51.569499 28382 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9: Loaded TSK: 0
I20260925 22:54:51.570145 28382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:54:51.600492 27837 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:54:51.603736 27837 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:54:51.606698 28372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.606404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.0.0.1:59720)
0925 22:54:51.606576 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.606634 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 22:54:51.606740 27664 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [term 2 FOLLOWER]: Refusing update from remote peer 1fc6c03407644ff6b318682cd66988a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 22:54:51.607288 28370 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 22:54:51.607414 28372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.607205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.0.0.1:59734)
0925 22:54:51.607314 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.607364 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 22:54:51.607532 27832 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 -> Peer c96778fffbfd40f186940fb51039e3ae (127.20.114.254:37819): Couldn't send request to peer c96778fffbfd40f186940fb51039e3ae. Status: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 22:54:51.608707 28375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fc6c03407644ff6b318682cd66988a9. Latest consensus state: current_term: 2 leader_uuid: "1fc6c03407644ff6b318682cd66988a9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: false } } }
I20260925 22:54:51.608856 28375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:54:51.609936 28370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "1fc6c03407644ff6b318682cd66988a9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: false } } }
I20260925 22:54:51.610055 28370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:51.610291 28375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "1fc6c03407644ff6b318682cd66988a9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96778fffbfd40f186940fb51039e3ae" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37819 } } peers { permanent_uuid: "88d59d46332e40e1af313320f228c2dc" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 35653 } attrs { promote: false } } peers { permanent_uuid: "1fc6c03407644ff6b318682cd66988a9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45987 } attrs { promote: false } } }
I20260925 22:54:51.610411 28375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88d59d46332e40e1af313320f228c2dc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:54:51.620936 28372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.615844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:34885 (local address 127.0.0.1:57896)
0925 22:54:51.615999 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.616016 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:51.616092 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:51.617338 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:51.617340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:51.617358 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:51.617445 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:51.617450 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.618276 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.618280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:51.619667 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:51.619677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.620548 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.620551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:51.620626 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:51.620644 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 22:54:51.620706 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 22:54:51.620763 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":103}
I20260925 22:54:51.621106 28388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.615981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57896 (local address 127.20.114.195:34885)
0925 22:54:51.617007 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:51.617012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:51.617026 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:54:51.617081 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:51.617087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:51.617139 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:51.617333 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:51.617596 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.618168 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.619818 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.620446 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.620565 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:51.620618 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:51.620856 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 22:54:51.620962 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":942,"thread_start_us":68,"threads_started":1}
I20260925 22:54:51.621141 28371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.616397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:43103 (local address 127.0.0.1:59452)
0925 22:54:51.616550 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.616564 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:51.616648 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:51.617202 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:51.617205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:51.617222 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:51.617316 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:51.617324 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.618335 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.618338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:51.619684 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:51.619689 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.620716 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.620720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:51.620809 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:51.620830 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:54:51.620933 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 22:54:51.621015 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":124}
I20260925 22:54:51.621157 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.616532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.20.114.194:43103)
0925 22:54:51.616936 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:51.616942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:51.616960 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:54:51.617016 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:51.617019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:51.617075 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:51.617193 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:51.617439 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.618249 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.619963 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.620617 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.620736 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:51.620783 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:51.620964 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 22:54:51.621041 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":91,"threads_started":1}
I20260925 22:54:51.621407 28390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.616536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46450 (local address 127.20.114.193:33393)
0925 22:54:51.616779 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:51.616784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:51.617224 (+ 440us) server_negotiation.cc:407] Connection header received
0925 22:54:51.617291 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:51.617295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:51.617349 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:51.617419 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:51.617785 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.618503 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.619629 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.620236 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.620369 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:51.620425 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:51.621168 (+ 743us) server_negotiation.cc:299] Negotiation successful
0925 22:54:51.621268 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
I20260925 22:54:51.620796 28391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.616397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:33393 (local address 127.0.0.1:46450)
0925 22:54:51.617041 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.617054 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:51.617136 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:51.617526 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:51.617528 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:51.617566 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:51.617656 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:51.617662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.618605 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.618610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:51.619501 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:51.619508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.620353 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.620356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:51.620442 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:51.620466 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:54:51.620542 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:54:51.620613 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":320,"thread_start_us":98,"threads_started":1}
I20260925 22:54:51.621588 28231 tablet_service.cc:1467] Processing CreateTablet for tablet 586fa5ac383b4f57a9a97bfe9c7d0493 (DEFAULT_TABLE table=test-table [id=879c047a262349bb8bc2b803ec0eb765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:51.621865 28231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 586fa5ac383b4f57a9a97bfe9c7d0493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:51.622390 27958 tablet_service.cc:1467] Processing CreateTablet for tablet 586fa5ac383b4f57a9a97bfe9c7d0493 (DEFAULT_TABLE table=test-table [id=879c047a262349bb8bc2b803ec0eb765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:51.622406 28095 tablet_service.cc:1467] Processing CreateTablet for tablet 586fa5ac383b4f57a9a97bfe9c7d0493 (DEFAULT_TABLE table=test-table [id=879c047a262349bb8bc2b803ec0eb765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:51.622658 27958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 586fa5ac383b4f57a9a97bfe9c7d0493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:51.622716 28095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 586fa5ac383b4f57a9a97bfe9c7d0493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:51.624508 28392 tablet_bootstrap.cc:492] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721: Bootstrap starting.
I20260925 22:54:51.624894 28393 tablet_bootstrap.cc:492] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21: Bootstrap starting.
I20260925 22:54:51.625568 28392 tablet_bootstrap.cc:654] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:51.625980 28393 tablet_bootstrap.cc:654] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:51.626613 28394 tablet_bootstrap.cc:492] T 586fa5ac383b4f57a9a97bfe9c7d0493 P c01ee4ba045949eebcc2660e769f7b6f: Bootstrap starting.
I20260925 22:54:51.626744 28392 tablet_bootstrap.cc:492] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721: No bootstrap required, opened a new log
I20260925 22:54:51.626830 28392 ts_tablet_manager.cc:1397] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:51.627112 28393 tablet_bootstrap.cc:492] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21: No bootstrap required, opened a new log
I20260925 22:54:51.627179 28393 ts_tablet_manager.cc:1397] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:51.627295 28392 raft_consensus.cc:348] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a341f62c9efa4a52995a7670f9e61721" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 34885 } } peers { permanent_uuid: "c01ee4ba045949eebcc2660e769f7b6f" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 33393 } } peers { permanent_uuid: "92bfd5c259204d5994cfd0029b038c21" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43103 } }
I20260925 22:54:51.627415 28392 raft_consensus.cc:374] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:51.627458 28392 raft_consensus.cc:729] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a341f62c9efa4a52995a7670f9e61721, State: Initialized, Role: FOLLOWER
I20260925 22:54:51.627585 28392 consensus_queue.cc:260] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a341f62c9efa4a52995a7670f9e61721" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 34885 } } peers { permanent_uuid: "c01ee4ba045949eebcc2660e769f7b6f" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 33393 } } peers { permanent_uuid: "92bfd5c259204d5994cfd0029b038c21" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43103 } }
I20260925 22:54:51.627614 28393 raft_consensus.cc:348] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a341f62c9efa4a52995a7670f9e61721" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 34885 } } peers { permanent_uuid: "c01ee4ba045949eebcc2660e769f7b6f" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 33393 } } peers { permanent_uuid: "92bfd5c259204d5994cfd0029b038c21" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43103 } }
I20260925 22:54:51.627697 28393 raft_consensus.cc:374] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:51.627753 28393 raft_consensus.cc:729] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92bfd5c259204d5994cfd0029b038c21, State: Initialized, Role: FOLLOWER
I20260925 22:54:51.627918 28392 ts_tablet_manager.cc:1428] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:51.627883 28393 consensus_queue.cc:260] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a341f62c9efa4a52995a7670f9e61721" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 34885 } } peers { permanent_uuid: "c01ee4ba045949eebcc2660e769f7b6f" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 33393 } } peers { permanent_uuid: "92bfd5c259204d5994cfd0029b038c21" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43103 } }
I20260925 22:54:51.628180 28393 ts_tablet_manager.cc:1428] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:51.628759 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.628374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.20.114.195:34317)
0925 22:54:51.628596 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.628656 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":94,"threads_started":1}
I20260925 22:54:51.630393 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.628434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.20.114.194:39239)
0925 22:54:51.630226 (+ 1792us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.630282 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1728,"thread_start_us":102,"threads_started":1}
I20260925 22:54:51.630656 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.629963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.20.114.195:58079)
0925 22:54:51.630534 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.630591 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504}
I20260925 22:54:51.631378 28297 heartbeater.cc:503] Master 127.20.114.252:45987 was elected leader, sending a full tablet report...
I20260925 22:54:51.631433 28394 tablet_bootstrap.cc:654] T 586fa5ac383b4f57a9a97bfe9c7d0493 P c01ee4ba045949eebcc2660e769f7b6f: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:51.631516 28161 heartbeater.cc:503] Master 127.20.114.252:45987 was elected leader, sending a full tablet report...
I20260925 22:54:51.632037 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.630889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.20.114.194:49677)
0925 22:54:51.631887 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.631939 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":934,"thread_start_us":99,"threads_started":1}
I20260925 22:54:51.632622 28394 tablet_bootstrap.cc:492] T 586fa5ac383b4f57a9a97bfe9c7d0493 P c01ee4ba045949eebcc2660e769f7b6f: No bootstrap required, opened a new log
I20260925 22:54:51.632737 28394 ts_tablet_manager.cc:1397] T 586fa5ac383b4f57a9a97bfe9c7d0493 P c01ee4ba045949eebcc2660e769f7b6f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260925 22:54:51.633176 28394 raft_consensus.cc:348] T 586fa5ac383b4f57a9a97bfe9c7d0493 P c01ee4ba045949eebcc2660e769f7b6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a341f62c9efa4a52995a7670f9e61721" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 34885 } } peers { permanent_uuid: "c01ee4ba045949eebcc2660e769f7b6f" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 33393 } } peers { permanent_uuid: "92bfd5c259204d5994cfd0029b038c21" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43103 } }
I20260925 22:54:51.633291 28394 raft_consensus.cc:374] T 586fa5ac383b4f57a9a97bfe9c7d0493 P c01ee4ba045949eebcc2660e769f7b6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:51.633334 28394 raft_consensus.cc:729] T 586fa5ac383b4f57a9a97bfe9c7d0493 P c01ee4ba045949eebcc2660e769f7b6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c01ee4ba045949eebcc2660e769f7b6f, State: Initialized, Role: FOLLOWER
I20260925 22:54:51.633473 28394 consensus_queue.cc:260] T 586fa5ac383b4f57a9a97bfe9c7d0493 P c01ee4ba045949eebcc2660e769f7b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a341f62c9efa4a52995a7670f9e61721" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 34885 } } peers { permanent_uuid: "c01ee4ba045949eebcc2660e769f7b6f" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 33393 } } peers { permanent_uuid: "92bfd5c259204d5994cfd0029b038c21" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43103 } }
I20260925 22:54:51.633807 28394 ts_tablet_manager.cc:1428] T 586fa5ac383b4f57a9a97bfe9c7d0493 P c01ee4ba045949eebcc2660e769f7b6f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:51.634678 28404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.634221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.20.114.193:44055)
0925 22:54:51.634457 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.634572 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":115,"threads_started":1}
I20260925 22:54:51.635614 28404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.635389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.20.114.193:47723)
0925 22:54:51.635497 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.635551 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 22:54:51.635643 28024 heartbeater.cc:503] Master 127.20.114.252:45987 was elected leader, sending a full tablet report...
I20260925 22:54:51.703925 28398 raft_consensus.cc:482] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:54:51.704062 28398 raft_consensus.cc:504] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a341f62c9efa4a52995a7670f9e61721" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 34885 } } peers { permanent_uuid: "c01ee4ba045949eebcc2660e769f7b6f" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 33393 } } peers { permanent_uuid: "92bfd5c259204d5994cfd0029b038c21" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43103 } }
I20260925 22:54:51.705135 28398 leader_election.cc:288] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a341f62c9efa4a52995a7670f9e61721 (127.20.114.195:34885), c01ee4ba045949eebcc2660e769f7b6f (127.20.114.193:33393)
I20260925 22:54:51.707808 28397 raft_consensus.cc:482] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:54:51.707930 28397 raft_consensus.cc:504] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a341f62c9efa4a52995a7670f9e61721" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 34885 } } peers { permanent_uuid: "c01ee4ba045949eebcc2660e769f7b6f" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 33393 } } peers { permanent_uuid: "92bfd5c259204d5994cfd0029b038c21" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43103 } }
I20260925 22:54:51.709079 28390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.705493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:33681 (local address 127.20.114.193:33393)
0925 22:54:51.705616 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:51.705620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:51.705637 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:54:51.705810 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:51.705817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:51.705871 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:51.705945 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:51.706263 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.706799 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.708036 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.708474 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.708546 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:51.708588 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:51.708893 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 22:54:51.708964 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:54:51.709147 28397 leader_election.cc:288] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c01ee4ba045949eebcc2660e769f7b6f (127.20.114.193:33393), 92bfd5c259204d5994cfd0029b038c21 (127.20.114.194:43103)
I20260925 22:54:51.709316 28388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.705485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:41471 (local address 127.20.114.195:34885)
0925 22:54:51.705624 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:51.705628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:51.705742 (+ 114us) server_negotiation.cc:407] Connection header received
0925 22:54:51.705886 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:51.705888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:51.705950 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:51.706024 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:51.706310 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.706805 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.708051 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.708598 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.708683 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:51.708726 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:51.709099 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 22:54:51.709193 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:54:51.709481 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.705384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:33393 (local address 127.20.114.194:33681)
0925 22:54:51.705515 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.705532 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:51.705658 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:51.705980 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:51.705986 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:51.706005 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:51.706145 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:51.706156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.706921 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.706925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:51.707878 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:51.707886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.708681 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.708684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:51.708752 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:51.708766 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 22:54:51.708854 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 22:54:51.708942 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":5}
I20260925 22:54:51.709677 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.705384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:34885 (local address 127.20.114.194:41471)
0925 22:54:51.705636 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.705650 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:51.705785 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:51.706059 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:51.706064 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:51.706081 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:51.706179 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:51.706184 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.706918 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.706922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:51.707910 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:51.707921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.708819 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.708823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:51.708954 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:51.708979 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:54:51.709417 (+ 438us) client_negotiation.cc:240] Negotiation successful
0925 22:54:51.709527 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":186}
I20260925 22:54:51.710172 28251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586fa5ac383b4f57a9a97bfe9c7d0493" candidate_uuid: "92bfd5c259204d5994cfd0029b038c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a341f62c9efa4a52995a7670f9e61721" is_pre_election: true
I20260925 22:54:51.710311 28251 raft_consensus.cc:2454] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92bfd5c259204d5994cfd0029b038c21 in term 0.
I20260925 22:54:51.710408 27978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586fa5ac383b4f57a9a97bfe9c7d0493" candidate_uuid: "92bfd5c259204d5994cfd0029b038c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c01ee4ba045949eebcc2660e769f7b6f" is_pre_election: true
I20260925 22:54:51.710659 27978 raft_consensus.cc:2454] T 586fa5ac383b4f57a9a97bfe9c7d0493 P c01ee4ba045949eebcc2660e769f7b6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92bfd5c259204d5994cfd0029b038c21 in term 0.
I20260925 22:54:51.710830 28050 leader_election.cc:302] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92bfd5c259204d5994cfd0029b038c21, a341f62c9efa4a52995a7670f9e61721; no voters:
I20260925 22:54:51.711082 28398 raft_consensus.cc:2781] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:54:51.711138 28398 raft_consensus.cc:482] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:54:51.711171 28398 raft_consensus.cc:3037] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:51.712430 28398 raft_consensus.cc:504] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a341f62c9efa4a52995a7670f9e61721" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 34885 } } peers { permanent_uuid: "c01ee4ba045949eebcc2660e769f7b6f" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 33393 } } peers { permanent_uuid: "92bfd5c259204d5994cfd0029b038c21" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43103 } }
I20260925 22:54:51.713095 27978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586fa5ac383b4f57a9a97bfe9c7d0493" candidate_uuid: "92bfd5c259204d5994cfd0029b038c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c01ee4ba045949eebcc2660e769f7b6f"
I20260925 22:54:51.713229 27978 raft_consensus.cc:3037] T 586fa5ac383b4f57a9a97bfe9c7d0493 P c01ee4ba045949eebcc2660e769f7b6f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:51.713034 28398 leader_election.cc:288] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21 [CANDIDATE]: Term 1 election: Requested vote from peers a341f62c9efa4a52995a7670f9e61721 (127.20.114.195:34885), c01ee4ba045949eebcc2660e769f7b6f (127.20.114.193:33393)
I20260925 22:54:51.713389 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.709301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:33393 (local address 127.20.114.195:40717)
0925 22:54:51.709419 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.709432 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:51.709539 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:51.710310 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:51.710313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:51.710331 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:51.710430 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:51.710439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.711354 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.711363 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:51.712134 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:51.712142 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.713006 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.713010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:51.713114 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:51.713135 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:54:51.713203 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 22:54:51.713266 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 22:54:51.713618 28251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586fa5ac383b4f57a9a97bfe9c7d0493" candidate_uuid: "92bfd5c259204d5994cfd0029b038c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a341f62c9efa4a52995a7670f9e61721"
I20260925 22:54:51.713761 28251 raft_consensus.cc:3037] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:51.713979 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.709808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:40721 (local address 127.20.114.194:43103)
0925 22:54:51.709897 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:51.709901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:51.710083 (+ 182us) server_negotiation.cc:407] Connection header received
0925 22:54:51.710250 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:51.710255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:51.710299 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:51.710343 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:51.710724 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.711375 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.712737 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.713340 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.713448 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:51.713495 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:51.713808 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 22:54:51.713872 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 22:54:51.714370 28405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.709672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:43103 (local address 127.20.114.195:40721)
0925 22:54:51.710003 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.710016 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:51.710146 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:51.710460 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:51.710475 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:51.710491 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:51.710596 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:51.710602 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.711523 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.711526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:51.712611 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:51.712621 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.713575 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.713579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:51.713670 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:51.713691 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:54:51.714158 (+ 467us) client_negotiation.cc:240] Negotiation successful
0925 22:54:51.714234 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":126,"threads_started":1}
I20260925 22:54:51.714512 27978 raft_consensus.cc:2454] T 586fa5ac383b4f57a9a97bfe9c7d0493 P c01ee4ba045949eebcc2660e769f7b6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92bfd5c259204d5994cfd0029b038c21 in term 1.
I20260925 22:54:51.714856 28048 leader_election.cc:302] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92bfd5c259204d5994cfd0029b038c21, c01ee4ba045949eebcc2660e769f7b6f; no voters:
I20260925 22:54:51.714887 28115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586fa5ac383b4f57a9a97bfe9c7d0493" candidate_uuid: "a341f62c9efa4a52995a7670f9e61721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92bfd5c259204d5994cfd0029b038c21" is_pre_election: true
I20260925 22:54:51.714939 28251 raft_consensus.cc:2454] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92bfd5c259204d5994cfd0029b038c21 in term 1.
I20260925 22:54:51.715058 28115 raft_consensus.cc:2379] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a341f62c9efa4a52995a7670f9e61721 in current term 1: Already voted for candidate 92bfd5c259204d5994cfd0029b038c21 in this term.
I20260925 22:54:51.715160 28398 raft_consensus.cc:2781] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:51.715411 28398 raft_consensus.cc:686] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21 [term 1 LEADER]: Becoming Leader. State: Replica: 92bfd5c259204d5994cfd0029b038c21, State: Running, Role: LEADER
I20260925 22:54:51.715504 28390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.709433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:40717 (local address 127.20.114.193:33393)
0925 22:54:51.710027 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:51.710031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:51.710047 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:54:51.710098 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:51.710101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:51.710142 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:51.710215 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:51.710561 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.711232 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.712265 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.712764 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.712869 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:51.712929 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:51.715331 (+ 2402us) server_negotiation.cc:299] Negotiation successful
0925 22:54:51.715402 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530}
I20260925 22:54:51.715847 28398 consensus_queue.cc:237] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a341f62c9efa4a52995a7670f9e61721" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 34885 } } peers { permanent_uuid: "c01ee4ba045949eebcc2660e769f7b6f" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 33393 } } peers { permanent_uuid: "92bfd5c259204d5994cfd0029b038c21" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43103 } }
I20260925 22:54:51.715961 27978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586fa5ac383b4f57a9a97bfe9c7d0493" candidate_uuid: "a341f62c9efa4a52995a7670f9e61721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c01ee4ba045949eebcc2660e769f7b6f" is_pre_election: true
I20260925 22:54:51.716141 27978 raft_consensus.cc:2379] T 586fa5ac383b4f57a9a97bfe9c7d0493 P c01ee4ba045949eebcc2660e769f7b6f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a341f62c9efa4a52995a7670f9e61721 in current term 1: Already voted for candidate 92bfd5c259204d5994cfd0029b038c21 in this term.
I20260925 22:54:51.716305 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.716050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.20.114.194:42499)
0925 22:54:51.716196 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.716249 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 22:54:51.716572 28184 leader_election.cc:302] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a341f62c9efa4a52995a7670f9e61721; no voters: 92bfd5c259204d5994cfd0029b038c21, c01ee4ba045949eebcc2660e769f7b6f
I20260925 22:54:51.716811 28397 raft_consensus.cc:2726] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 22:54:51.718605 27839 catalog_manager.cc:5495] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92bfd5c259204d5994cfd0029b038c21 (127.20.114.194). New cstate: current_term: 1 leader_uuid: "92bfd5c259204d5994cfd0029b038c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a341f62c9efa4a52995a7670f9e61721" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 34885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c01ee4ba045949eebcc2660e769f7b6f" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92bfd5c259204d5994cfd0029b038c21" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 43103 } health_report { overall_health: HEALTHY } } }
I20260925 22:54:51.747558 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.743889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59468 (local address 127.20.114.194:43103)
0925 22:54:51.744032 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:51.744037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:51.744052 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:54:51.744200 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:51.744202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:51.744255 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:51.744365 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:51.744601 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.745181 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.746103 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:51.746322 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:51.746431 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:51.746713 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:51.747171 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:51.747295 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:51.747364 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 22:54:51.747413 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:54:51.750461 28251 raft_consensus.cc:1261] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721 [term 1 FOLLOWER]: Refusing update from remote peer 92bfd5c259204d5994cfd0029b038c21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:51.750808 27978 raft_consensus.cc:1261] T 586fa5ac383b4f57a9a97bfe9c7d0493 P c01ee4ba045949eebcc2660e769f7b6f [term 1 FOLLOWER]: Refusing update from remote peer 92bfd5c259204d5994cfd0029b038c21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:51.751006 28406 consensus_queue.cc:1035] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a341f62c9efa4a52995a7670f9e61721" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 34885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:51.751370 28398 consensus_queue.cc:1035] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c01ee4ba045949eebcc2660e769f7b6f" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 33393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:51.752957 28405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.752627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.20.114.195:45115)
0925 22:54:51.752804 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.752881 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 22:54:51.753203 28404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.752980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.20.114.193:35411)
0925 22:54:51.753082 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.753135 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 22:54:51.756482 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.756137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.20.114.194:59355)
0925 22:54:51.756356 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.756408 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260925 22:54:51.757158 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.756908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.20.114.194:38545)
0925 22:54:51.757038 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.757092 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 22:54:51.758008 28405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.757546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.20.114.195:54495)
0925 22:54:51.757884 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.757937 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235}
I20260925 22:54:51.758019 28412 mvcc.cc:205] Tried to move back new op lower bound from 7333383748604514304 to 7333383748471349248. Current Snapshot: MvccSnapshot[applied={T|T < 7333383748604514304}]
I20260925 22:54:51.758589 28404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:51.758284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37819 (local address 127.20.114.193:50953)
0925 22:54:51.758478 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.758527 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260925 22:54:51.759145 28410 mvcc.cc:205] Tried to move back new op lower bound from 7333383748604514304 to 7333383748471349248. Current Snapshot: MvccSnapshot[applied={T|T < 7333383748604514304 or (T in {7333383748604514304})}]
I20260925 22:54:51.766326 27839 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:51.766521 27839 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:51.770821 27839 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1fc6c03407644ff6b318682cd66988a9: Sending DeleteTablet for 3 replicas of tablet 586fa5ac383b4f57a9a97bfe9c7d0493
I20260925 22:54:51.771634 27958 tablet_service.cc:1514] Processing DeleteTablet for tablet 586fa5ac383b4f57a9a97bfe9c7d0493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:51 UTC) from {username='oryx', principal='oryx/127.20.114.252@KRBTEST.COM'} at 127.0.0.1:46450
I20260925 22:54:51.771720 28095 tablet_service.cc:1514] Processing DeleteTablet for tablet 586fa5ac383b4f57a9a97bfe9c7d0493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:51 UTC) from {username='oryx', principal='oryx/127.20.114.252@KRBTEST.COM'} at 127.0.0.1:59452
I20260925 22:54:51.772013 28421 tablet_replica.cc:330] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21: stopping tablet replica
I20260925 22:54:51.772135 28421 raft_consensus.cc:2229] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:54:51.772293 28231 tablet_service.cc:1514] Processing DeleteTablet for tablet 586fa5ac383b4f57a9a97bfe9c7d0493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:51 UTC) from {username='oryx', principal='oryx/127.20.114.252@KRBTEST.COM'} at 127.0.0.1:57896
I20260925 22:54:51.772450 28421 raft_consensus.cc:2258] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:51.773425 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 27799
I20260925 22:54:51.774318 28422 tablet_replica.cc:330] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721: stopping tablet replica
I20260925 22:54:51.774554 28422 raft_consensus.cc:2229] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:51.774784 28422 raft_consensus.cc:2258] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:51.775640 28421 ts_tablet_manager.cc:1905] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:51.775790 28422 ts_tablet_manager.cc:1905] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:51.778206 28421 ts_tablet_manager.cc:1918] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:54:51.778206 28422 ts_tablet_manager.cc:1918] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:54:51.778287 28421 log.cc:1196] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/586fa5ac383b4f57a9a97bfe9c7d0493
I20260925 22:54:51.778404 28422 log.cc:1196] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/586fa5ac383b4f57a9a97bfe9c7d0493
I20260925 22:54:51.778570 28421 ts_tablet_manager.cc:1939] T 586fa5ac383b4f57a9a97bfe9c7d0493 P 92bfd5c259204d5994cfd0029b038c21: Deleting consensus metadata
I20260925 22:54:51.778787 28422 ts_tablet_manager.cc:1939] T 586fa5ac383b4f57a9a97bfe9c7d0493 P a341f62c9efa4a52995a7670f9e61721: Deleting consensus metadata
I20260925 22:54:51.779300 27831 catalog_manager.cc:4841] TS 92bfd5c259204d5994cfd0029b038c21 (127.20.114.194:43103): tablet 586fa5ac383b4f57a9a97bfe9c7d0493 (table test-table [id=879c047a262349bb8bc2b803ec0eb765]) successfully deleted
I20260925 22:54:51.779500 27831 catalog_manager.cc:4841] TS a341f62c9efa4a52995a7670f9e61721 (127.20.114.195:34885): tablet 586fa5ac383b4f57a9a97bfe9c7d0493 (table test-table [id=879c047a262349bb8bc2b803ec0eb765]) successfully deleted
W20260925 22:54:51.783591 28391 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:51.782545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:33393 (local address 127.0.0.1:46456)
0925 22:54:51.782712 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:51.782727 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:51.782807 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:51.783468 (+ 661us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.193:33393: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":87}
W20260925 22:54:51.783809 27829 catalog_manager.cc:4584] TS c01ee4ba045949eebcc2660e769f7b6f (127.20.114.193:33393): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 586fa5ac383b4f57a9a97bfe9c7d0493: Network error: Client connection negotiation failed: client connection to 127.20.114.193:33393: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 22:54:51.784328 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 28032
I20260925 22:54:51.790153 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 28168
I20260925 22:54:51.796618 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 27518
I20260925 22:54:51.802399 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 27621
2026-09-25T22:54:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5580 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:54:51 dist-test-slave-x1lh krb5kdc[28425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:54:51 dist-test-slave-x1lh krb5kdc[28425](info): set up 2 sockets
Sep 25 22:54:51 dist-test-slave-x1lh krb5kdc[28425](info): commencing operation
krb5kdc: starting...
W20260925 22:54:52.837457 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:54:52 dist-test-slave-x1lh krb5kdc[28425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:54:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:54:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.254.keytab.
Entry for principal oryx/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.254.keytab.
I20260925 22:54:52.957742 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:41055
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.254.keytab
--principal=oryx/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:41301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:41055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:53.067607 28438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:53.067869 28438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:53.067909 28438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:53.071946 28438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:53.072000 28438 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 22:54:53.072017 28438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:53.072037 28438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:53.072060 28438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:53.072075 28438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:53.077370 28438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:41301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:41055
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.254.keytab
--principal=oryx/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:41055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:53.078442 28438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:53.079464 28438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:54:53.087888 28438 server_base.cc:1034] running on GCE node
W20260925 22:54:53.087862 28444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:53.087854 28446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:53.088073 28443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:53.088634 28438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:53.089783 28438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:53.090969 28438 hybrid_clock.cc:648] HybridClock initialized: now 1790376893090960 us; error 31 us; skew 500 ppm
Sep 25 22:54:53 dist-test-slave-x1lh krb5kdc[28425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376893, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:53.093639 28438 init.cc:377] Logged in from keytab as oryx/127.20.114.254@KRBTEST.COM (short username oryx)
I20260925 22:54:53.094456 28438 webserver.cc:460] Webserver started at http://127.20.114.254:42391/ using document root <none> and password file <none>
I20260925 22:54:53.095001 28438 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:53.095091 28438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:53.095328 28438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:53.097378 28438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "de923d24b4aa4c2bad4faf9b72a86537"
format_stamp: "Formatted at 2026-09-25 22:54:53 on dist-test-slave-x1lh"
server_key: "8c47522670b0428ddbc9df442a81b8b5"
server_key_iv: "c882f3de7fdc99dd360060d366252f1d"
server_key_version: "encryptionkey@0"
I20260925 22:54:53.097935 28438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "de923d24b4aa4c2bad4faf9b72a86537"
format_stamp: "Formatted at 2026-09-25 22:54:53 on dist-test-slave-x1lh"
server_key: "8c47522670b0428ddbc9df442a81b8b5"
server_key_iv: "c882f3de7fdc99dd360060d366252f1d"
server_key_version: "encryptionkey@0"
I20260925 22:54:53.101522 28438 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:54:53.103859 28453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:53.105127 28438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:53.105254 28438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "de923d24b4aa4c2bad4faf9b72a86537"
format_stamp: "Formatted at 2026-09-25 22:54:53 on dist-test-slave-x1lh"
server_key: "8c47522670b0428ddbc9df442a81b8b5"
server_key_iv: "c882f3de7fdc99dd360060d366252f1d"
server_key_version: "encryptionkey@0"
I20260925 22:54:53.105362 28438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:53.116483 28438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:53.119066 28438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:53.119268 28438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:53.127252 28438 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:41055
I20260925 22:54:53.127270 28515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:41055 every 8 connection(s)
I20260925 22:54:53.128198 28438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:54:53.131017 28516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:53.133718 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 28438
I20260925 22:54:53.133831 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:54:53.134117 20939 external_mini_cluster.cc:1428] Setting key a66d780c5a9a68a7f1e3f56e00ab929f
I20260925 22:54:53.137250 28516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537: Bootstrap starting.
I20260925 22:54:53.139430 28516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:53.140178 28516 log.cc:824] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:53.142134 28516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537: No bootstrap required, opened a new log
Sep 25 22:54:53 dist-test-slave-x1lh krb5kdc[28425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM
I20260925 22:54:53.145812 28516 raft_consensus.cc:348] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de923d24b4aa4c2bad4faf9b72a86537" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 41055 } }
I20260925 22:54:53.146051 28516 raft_consensus.cc:374] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:53.146150 28516 raft_consensus.cc:729] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de923d24b4aa4c2bad4faf9b72a86537, State: Initialized, Role: FOLLOWER
I20260925 22:54:53.146720 28516 consensus_queue.cc:260] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de923d24b4aa4c2bad4faf9b72a86537" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 41055 } }
I20260925 22:54:53.146893 28516 raft_consensus.cc:388] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:54:53.146967 28516 raft_consensus.cc:482] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:54:53.147068 28516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:53.147799 28519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:53.135632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56174 (local address 127.20.114.254:41055)
0925 22:54:53.136069 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:53.136084 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:53.136119 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:54:53.136712 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:53.136727 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:53.137115 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:53.137438 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:53.138267 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.139243 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.140150 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.140649 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.140814 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:53.143226 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:53.143231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:53.143243 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:53.143268 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:53.144957 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:53.145374 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:53.145380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:53.145390 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:53.145494 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:53.145767 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:53.145771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:53.145773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:53.146236 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:53.146449 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:53.146724 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 22:54:53.147016 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":130,"threads_started":1}
I20260925 22:54:53.147998 28516 raft_consensus.cc:504] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de923d24b4aa4c2bad4faf9b72a86537" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 41055 } }
I20260925 22:54:53.148348 28516 leader_election.cc:302] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de923d24b4aa4c2bad4faf9b72a86537; no voters:
I20260925 22:54:53.148685 28516 leader_election.cc:288] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:54:53.148844 28521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:53.149143 28521 raft_consensus.cc:686] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537 [term 1 LEADER]: Becoming Leader. State: Replica: de923d24b4aa4c2bad4faf9b72a86537, State: Running, Role: LEADER
I20260925 22:54:53.149434 28521 consensus_queue.cc:237] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de923d24b4aa4c2bad4faf9b72a86537" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 41055 } }
I20260925 22:54:53.149983 28516 sys_catalog.cc:564] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:53.150591 28521 sys_catalog.cc:455] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de923d24b4aa4c2bad4faf9b72a86537. Latest consensus state: current_term: 1 leader_uuid: "de923d24b4aa4c2bad4faf9b72a86537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de923d24b4aa4c2bad4faf9b72a86537" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 41055 } } }
I20260925 22:54:53.150732 28521 sys_catalog.cc:458] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:53.151132 28522 sys_catalog.cc:455] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de923d24b4aa4c2bad4faf9b72a86537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de923d24b4aa4c2bad4faf9b72a86537" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 41055 } } }
I20260925 22:54:53.151276 28522 sys_catalog.cc:458] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:53.152348 28527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:54:53.154899 28527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:54:53.160496 28527 catalog_manager.cc:1347] Generated new cluster ID: 19cd6e9804944bc1aee446f6e86d87c7
I20260925 22:54:53.160567 28527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:53.175478 28527 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:54:53.176406 28527 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:53.180056 28527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537: Generated new TSK 0
I20260925 22:54:53.180716 28527 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.114.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.114.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.114.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.193.keytab.
Entry for principal oryx/127.20.114.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.193.keytab.
I20260925 22:54:53.216517 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:41055
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.193.keytab
--principal=oryx/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:41301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:53.346072 28542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:53.346345 28542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:53.346390 28542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:53.350589 28542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:53.350647 28542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:53.350688 28542 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:54:53.350757 28542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:54:53.356217 28542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:41301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.193.keytab
--principal=oryx/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.114.254:41055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:53.357288 28542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:53.358330 28542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:53.370223 28548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:53.370656 28547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:53.370951 28542 server_base.cc:1034] running on GCE node
W20260925 22:54:53.370209 28550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:53.371647 28542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:53.372339 28542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:53.373521 28542 hybrid_clock.cc:648] HybridClock initialized: now 1790376893373506 us; error 32 us; skew 500 ppm
Sep 25 22:54:53 dist-test-slave-x1lh krb5kdc[28425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376893, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:53.376137 28542 init.cc:377] Logged in from keytab as oryx/127.20.114.193@KRBTEST.COM (short username oryx)
I20260925 22:54:53.377035 28542 webserver.cc:460] Webserver started at http://127.20.114.193:44011/ using document root <none> and password file <none>
I20260925 22:54:53.377590 28542 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:53.377691 28542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:53.377919 28542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:53.380354 28542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "fdfa243493eb4df8bb1907dadb1ae03b"
format_stamp: "Formatted at 2026-09-25 22:54:53 on dist-test-slave-x1lh"
server_key: "5919d337f16b47b7ce8e1276611aad77"
server_key_iv: "da536f28eb989a7cb888b8322d1d9ad9"
server_key_version: "encryptionkey@0"
I20260925 22:54:53.380991 28542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "fdfa243493eb4df8bb1907dadb1ae03b"
format_stamp: "Formatted at 2026-09-25 22:54:53 on dist-test-slave-x1lh"
server_key: "5919d337f16b47b7ce8e1276611aad77"
server_key_iv: "da536f28eb989a7cb888b8322d1d9ad9"
server_key_version: "encryptionkey@0"
I20260925 22:54:53.384682 28542 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:54:53.387032 28557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:53.388162 28542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:53.388288 28542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "fdfa243493eb4df8bb1907dadb1ae03b"
format_stamp: "Formatted at 2026-09-25 22:54:53 on dist-test-slave-x1lh"
server_key: "5919d337f16b47b7ce8e1276611aad77"
server_key_iv: "da536f28eb989a7cb888b8322d1d9ad9"
server_key_version: "encryptionkey@0"
I20260925 22:54:53.388401 28542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:53.406136 28542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:53.409116 28542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:53.409332 28542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:53.410768 28542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:53.410837 28542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:53.410909 28542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:53.410959 28542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:54:53 dist-test-slave-x1lh krb5kdc[28425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376893, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM
I20260925 22:54:53.421473 28542 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:45563
I20260925 22:54:53.421725 28672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:45563 every 8 connection(s)
I20260925 22:54:53.422380 28542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:54:53.422979 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 28542
I20260925 22:54:53.423112 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:54:53.423436 20939 external_mini_cluster.cc:1428] Setting key 7333f91ddb416d9de4a4385c4b30875d
I20260925 22:54:53.425871 28519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:53.411663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:34801 (local address 127.20.114.254:41055)
0925 22:54:53.411806 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:53.411809 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:53.412904 (+ 1095us) server_negotiation.cc:407] Connection header received
0925 22:54:53.413995 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:53.413999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:53.414062 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:53.414167 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:53.415800 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.416659 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.418577 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.418820 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.418902 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:53.422035 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:53.422038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:53.422041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:53.422074 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:53.423777 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:53.424346 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:53.424351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:53.424353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:53.424447 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:53.424776 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:53.424779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:53.424781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:53.425022 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:53.425126 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:53.425592 (+ 466us) server_negotiation.cc:299] Negotiation successful
0925 22:54:53.425708 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:54:53.426856 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:53.412235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:41055 (local address 127.20.114.193:34801)
0925 22:54:53.412744 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:53.412786 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:53.413771 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:53.414404 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:53.414415 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:53.414912 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:53.415624 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:53.415646 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.416827 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.416832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:53.418434 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:53.418447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.418984 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.418989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:53.419213 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:53.419928 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:53.419954 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:53.421865 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:53.423891 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:53.423898 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:53.423913 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:53.424234 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:53.424524 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:53.424529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:53.424532 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:53.424667 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:53.425123 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:53.425129 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:53.425361 (+ 232us) client_negotiation.cc:769] Sending connection context
0925 22:54:53.425535 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 22:54:53.425843 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":120,"threads_started":1}
I20260925 22:54:53.428151 28673 heartbeater.cc:347] Connected to a master server at 127.20.114.254:41055
I20260925 22:54:53.428305 28673 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:53.428840 28673 heartbeater.cc:511] Master 127.20.114.254:41055 requested a full tablet report, sending...
I20260925 22:54:53.430612 28469 ts_manager.cc:194] Registered new tserver with Master: fdfa243493eb4df8bb1907dadb1ae03b (127.20.114.193:45563)
I20260925 22:54:53.431926 28469 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:34801
WARNING: no policy specified for oryx/127.20.114.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.114.194@KRBTEST.COM" created.
I20260925 22:54:53.439165 28519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:53.430284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:56645 (local address 127.20.114.254:41055)
0925 22:54:53.430415 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:53.430420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:53.430718 (+ 298us) server_negotiation.cc:407] Connection header received
0925 22:54:53.430818 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:53.430824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:53.430880 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:53.430972 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:53.431452 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.432288 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.433615 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.433918 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.434028 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:53.435136 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:53.435138 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:53.435140 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:53.435159 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:53.436429 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:53.438067 (+ 1638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:53.438070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:53.438071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:53.438144 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:53.438447 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:53.438450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:53.438451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:53.438601 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:53.438673 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:53.438969 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 22:54:53.439059 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 22:54:53.439386 28680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:53.430209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:41055 (local address 127.20.114.193:56645)
0925 22:54:53.430615 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:53.430633 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:53.430738 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:53.431000 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:53.431003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:53.431208 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:53.431328 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:53.431335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.432445 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.432452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:53.433487 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:53.433498 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.434043 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.434049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:53.434145 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:53.434649 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:53.434662 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:53.435029 (+ 367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:53.436529 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:53.436535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:53.436538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:53.436793 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:53.438243 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:53.438248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:53.438251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:53.438354 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:53.438692 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:53.438697 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:53.438800 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 22:54:53.438925 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 22:54:53.439082 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":229,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.114.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.194.keytab.
Entry for principal oryx/127.20.114.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.194.keytab.
I20260925 22:54:53.451730 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:41055
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.194.keytab
--principal=oryx/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:41301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:53.561393 28682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:53.561647 28682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:53.561691 28682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:53.566457 28682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:53.566514 28682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:53.566553 28682 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:54:53.566592 28682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:54:53.571857 28682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:41301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.194.keytab
--principal=oryx/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.114.254:41055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:53.572923 28682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:53.573918 28682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:53.583401 28688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:53.583606 28682 server_base.cc:1034] running on GCE node
W20260925 22:54:53.583413 28687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:53.583417 28690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:53.584105 28682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:53.584753 28682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:53.585939 28682 hybrid_clock.cc:648] HybridClock initialized: now 1790376893585909 us; error 49 us; skew 500 ppm
Sep 25 22:54:53 dist-test-slave-x1lh krb5kdc[28425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376893, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:53.588681 28682 init.cc:377] Logged in from keytab as oryx/127.20.114.194@KRBTEST.COM (short username oryx)
I20260925 22:54:53.589622 28682 webserver.cc:460] Webserver started at http://127.20.114.194:44681/ using document root <none> and password file <none>
I20260925 22:54:53.590201 28682 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:53.590296 28682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:53.590524 28682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:53.592686 28682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "cef6754891984e688cd7df6a8fc4f3a2"
format_stamp: "Formatted at 2026-09-25 22:54:53 on dist-test-slave-x1lh"
server_key: "a18df299dbcb61d9a9ee58448f834a68"
server_key_iv: "b919be7ed4196b36fe6868367ad3be8c"
server_key_version: "encryptionkey@0"
I20260925 22:54:53.593321 28682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "cef6754891984e688cd7df6a8fc4f3a2"
format_stamp: "Formatted at 2026-09-25 22:54:53 on dist-test-slave-x1lh"
server_key: "a18df299dbcb61d9a9ee58448f834a68"
server_key_iv: "b919be7ed4196b36fe6868367ad3be8c"
server_key_version: "encryptionkey@0"
I20260925 22:54:53.596952 28682 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:54:53.599414 28697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:53.600615 28682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:53.600754 28682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "cef6754891984e688cd7df6a8fc4f3a2"
format_stamp: "Formatted at 2026-09-25 22:54:53 on dist-test-slave-x1lh"
server_key: "a18df299dbcb61d9a9ee58448f834a68"
server_key_iv: "b919be7ed4196b36fe6868367ad3be8c"
server_key_version: "encryptionkey@0"
I20260925 22:54:53.600868 28682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:53.615757 28682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:53.618737 28682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:53.618937 28682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:53.620332 28682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:53.620404 28682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:53.620471 28682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:53.620519 28682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:54:53 dist-test-slave-x1lh krb5kdc[28425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376893, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM
I20260925 22:54:53.631992 28682 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:42959
I20260925 22:54:53.632071 28812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:42959 every 8 connection(s)
I20260925 22:54:53.633132 28682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:54:53.634415 28519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:53.621172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:44525 (local address 127.20.114.254:41055)
0925 22:54:53.621296 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:53.621302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:53.622343 (+ 1041us) server_negotiation.cc:407] Connection header received
0925 22:54:53.623326 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:53.623330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:53.623385 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:53.623442 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:53.625208 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.625758 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.627054 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.627334 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.627454 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:53.630337 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:53.630341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:53.630344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:53.630369 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:53.632009 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:53.632595 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:53.632598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:53.632599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:53.632675 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:53.633060 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:53.633063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:53.633064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:53.633453 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:53.633554 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:53.634130 (+ 576us) server_negotiation.cc:299] Negotiation successful
0925 22:54:53.634248 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:54:53.635141 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:53.621749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:41055 (local address 127.20.114.194:44525)
0925 22:54:53.622203 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:53.622244 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:53.623072 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:53.623883 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:53.623894 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:53.624389 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:53.625062 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:53.625083 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.625894 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.625897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:53.626944 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:53.626955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.627502 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.627506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:53.627727 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:53.628344 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:53.628369 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:53.630171 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:53.632158 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:53.632164 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:53.632177 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:53.632494 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:53.632775 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:53.632780 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:53.632781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:53.632964 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:53.633604 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:53.633610 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:53.633874 (+ 264us) client_negotiation.cc:769] Sending connection context
0925 22:54:53.634072 (+ 198us) client_negotiation.cc:240] Negotiation successful
0925 22:54:53.634382 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":124,"threads_started":1}
I20260925 22:54:53.636346 28813 heartbeater.cc:347] Connected to a master server at 127.20.114.254:41055
I20260925 22:54:53.636495 28813 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:53.637028 28813 heartbeater.cc:511] Master 127.20.114.254:41055 requested a full tablet report, sending...
I20260925 22:54:53.637898 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 28682
I20260925 22:54:53.638046 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:54:53.638316 20939 external_mini_cluster.cc:1428] Setting key 8ba7d8b3f1e14bf383c4726ea5a96042
I20260925 22:54:53.638746 28469 ts_manager.cc:194] Registered new tserver with Master: cef6754891984e688cd7df6a8fc4f3a2 (127.20.114.194:42959)
I20260925 22:54:53.639403 28469 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:44525
I20260925 22:54:53.646268 28519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:53.639055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:48879 (local address 127.20.114.254:41055)
0925 22:54:53.639183 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:53.639186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:53.639196 (+ 10us) server_negotiation.cc:407] Connection header received
0925 22:54:53.639222 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:53.639224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:53.639256 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:53.639339 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:53.639805 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.640514 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.641642 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.641904 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.641992 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:53.643101 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:53.643105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:53.643108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:53.643131 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:53.644576 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:53.645071 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:53.645076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:53.645078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:53.645157 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:53.645427 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:53.645432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:53.645433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:53.645638 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:53.645753 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:53.646000 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 22:54:53.646110 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:54:53.646301 28819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:53.638686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:41055 (local address 127.20.114.194:48879)
0925 22:54:53.638973 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:53.638987 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:53.639066 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:53.639356 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:53.639359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:53.639587 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:53.639695 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:53.639701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.640657 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.640660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:53.641534 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:53.641543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.641995 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.641999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:53.642085 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:53.642618 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:53.642631 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:53.642963 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:53.644708 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:53.644712 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:53.644714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:53.644978 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:53.645249 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:53.645253 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:53.645256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:53.645340 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:53.645742 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:53.645746 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:53.645842 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 22:54:53.645948 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 22:54:53.646098 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":28,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for oryx/127.20.114.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.114.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.114.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.195.keytab.
Entry for principal oryx/127.20.114.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.195.keytab.
I20260925 22:54:53.667683 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:41055
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.195.keytab
--principal=oryx/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:41301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:53.784097 28822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:53.784350 28822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:53.784394 28822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:53.788308 28822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:53.788378 28822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:53.788419 28822 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:54:53.788451 28822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:54:53.793877 28822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:41301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.195.keytab
--principal=oryx/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.114.254:41055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:53.794857 28822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:53.795832 28822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:53.805270 28827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:53.805431 28822 server_base.cc:1034] running on GCE node
W20260925 22:54:53.805279 28828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:53.805620 28830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:53.805994 28822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:53.806643 28822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:53.807827 28822 hybrid_clock.cc:648] HybridClock initialized: now 1790376893807819 us; error 27 us; skew 500 ppm
Sep 25 22:54:53 dist-test-slave-x1lh krb5kdc[28425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376893, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:53.810663 28822 init.cc:377] Logged in from keytab as oryx/127.20.114.195@KRBTEST.COM (short username oryx)
I20260925 22:54:53.811493 28822 webserver.cc:460] Webserver started at http://127.20.114.195:35847/ using document root <none> and password file <none>
I20260925 22:54:53.812077 28822 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:53.812170 28822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:53.812391 28822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:53.814435 28822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "3b5b7752b58b4603b857eb4acbae8ef2"
format_stamp: "Formatted at 2026-09-25 22:54:53 on dist-test-slave-x1lh"
server_key: "561e6edf663db269ef36da0f41344e8b"
server_key_iv: "86c4edeae2ca2399539d7b09239bdb88"
server_key_version: "encryptionkey@0"
I20260925 22:54:53.815022 28822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "3b5b7752b58b4603b857eb4acbae8ef2"
format_stamp: "Formatted at 2026-09-25 22:54:53 on dist-test-slave-x1lh"
server_key: "561e6edf663db269ef36da0f41344e8b"
server_key_iv: "86c4edeae2ca2399539d7b09239bdb88"
server_key_version: "encryptionkey@0"
I20260925 22:54:53.818620 28822 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260925 22:54:53.820958 28837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:53.822108 28822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:54:53.822237 28822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "3b5b7752b58b4603b857eb4acbae8ef2"
format_stamp: "Formatted at 2026-09-25 22:54:53 on dist-test-slave-x1lh"
server_key: "561e6edf663db269ef36da0f41344e8b"
server_key_iv: "86c4edeae2ca2399539d7b09239bdb88"
server_key_version: "encryptionkey@0"
I20260925 22:54:53.822356 28822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:53.836949 28822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:53.840104 28822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:53.840305 28822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:53.841692 28822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:53.841763 28822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:53.841833 28822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:53.841877 28822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:54:53 dist-test-slave-x1lh krb5kdc[28425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376893, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM
I20260925 22:54:53.852735 28822 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:36495
I20260925 22:54:53.852816 28952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:36495 every 8 connection(s)
I20260925 22:54:53.853900 28822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:54:53.856108 28519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:53.842562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:59141 (local address 127.20.114.254:41055)
0925 22:54:53.842685 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:53.842689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:53.843709 (+ 1020us) server_negotiation.cc:407] Connection header received
0925 22:54:53.844902 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:53.844906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:53.844969 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:53.845057 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:53.846659 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.847560 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.848898 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.849129 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.849257 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:53.852616 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:53.852621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:53.852627 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:53.852662 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:53.854290 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:53.854873 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:53.854876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:53.854878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:53.854959 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:53.855220 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:53.855222 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:53.855223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:53.855395 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:53.855459 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:53.855893 (+ 434us) server_negotiation.cc:299] Negotiation successful
0925 22:54:53.855988 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:54:53.856982 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:53.843081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:41055 (local address 127.20.114.195:59141)
0925 22:54:53.843540 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:53.843580 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:53.844650 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:53.845289 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:53.845301 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:53.845772 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:53.846483 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:53.846505 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.847679 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.847683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:53.848742 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:53.848751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.849308 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.849312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:53.849531 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:53.850679 (+ 1148us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:53.850713 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:53.852405 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:53.854423 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:53.854428 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:53.854442 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:53.854773 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:53.855037 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:53.855039 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:53.855040 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:53.855158 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:53.855483 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:53.855488 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:53.855697 (+ 209us) client_negotiation.cc:769] Sending connection context
0925 22:54:53.855841 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 22:54:53.856070 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":118,"threads_started":1}
I20260925 22:54:53.858148 28953 heartbeater.cc:347] Connected to a master server at 127.20.114.254:41055
I20260925 22:54:53.858291 28953 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:53.858841 28953 heartbeater.cc:511] Master 127.20.114.254:41055 requested a full tablet report, sending...
I20260925 22:54:53.859937 28469 ts_manager.cc:194] Registered new tserver with Master: 3b5b7752b58b4603b857eb4acbae8ef2 (127.20.114.195:36495)
I20260925 22:54:53.860731 28469 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:59141
I20260925 22:54:53.864125 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 28822
I20260925 22:54:53.864262 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:54:53.864550 20939 external_mini_cluster.cc:1428] Setting key 7c3444f54c179843c51cf0256b1e64a1
I20260925 22:54:53.866693 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:54:53.867795 28519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:53.860549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:52701 (local address 127.20.114.254:41055)
0925 22:54:53.860669 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:53.860673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:53.860834 (+ 161us) server_negotiation.cc:407] Connection header received
0925 22:54:53.860978 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:53.860981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:53.861034 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:53.861106 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:53.861609 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.862328 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.863102 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.863308 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.863387 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:53.864404 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:53.864415 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:53.864417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:53.864437 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:53.866181 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:53.866650 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:53.866653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:53.866655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:53.866758 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:53.867046 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:53.867049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:53.867050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:53.867224 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:53.867299 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:53.867587 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 22:54:53.867673 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 22:54:53.868094 28959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:53.860469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:41055 (local address 127.20.114.195:52701)
0925 22:54:53.860755 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:53.860767 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:53.860900 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:53.861139 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:53.861142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:53.861357 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:53.861496 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:53.861501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.862445 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.862447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:53.863001 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:53.863008 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.863404 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.863406 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:53.863469 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:53.863905 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:53.863915 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:53.864285 (+ 370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:53.866282 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:53.866285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:53.866287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:53.866537 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:53.866867 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:53.866870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:53.866872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:53.866958 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:53.867322 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:53.867324 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:53.867422 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 22:54:53.867543 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 22:54:53.867714 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
Sep 25 22:54:53 dist-test-slave-x1lh krb5kdc[28425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376893, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:54:53 dist-test-slave-x1lh krb5kdc[28425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:54:53 dist-test-slave-x1lh krb5kdc[28425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:54:53.890108 28519 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:53.884195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56182 (local address 127.20.114.254:41055)
0925 22:54:53.884303 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:53.884306 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:53.884319 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:54:53.884359 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:53.884362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:53.884401 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:53.884468 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:53.885236 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.885748 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.886629 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.886841 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.886918 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:53.889943 (+ 3025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56182: BlockingRecv error: recv got EOF from 127.0.0.1:56182 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 22:54:53 dist-test-slave-x1lh krb5kdc[28425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376893, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM
I20260925 22:54:53.899358 28519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:53.891559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56196 (local address 127.20.114.254:41055)
0925 22:54:53.891691 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:53.891695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:53.891815 (+ 120us) server_negotiation.cc:407] Connection header received
0925 22:54:53.892013 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:53.892015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:53.892054 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:53.892139 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:53.892719 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.893293 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.894059 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.894278 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.894386 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:53.896484 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:53.896492 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:53.896494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:53.896515 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:53.897791 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:53.898192 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:53.898197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:53.898198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:53.898276 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:53.898544 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:53.898548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:53.898550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:53.898725 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:53.898863 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:53.899092 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 22:54:53.899201 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:54:53.902130 28469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:54:53.905375 28469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:54:53.919276 28979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:53.914231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35088 (local address 127.20.114.195:36495)
0925 22:54:53.914649 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:53.914665 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:53.915004 (+ 339us) server_negotiation.cc:407] Connection header received
0925 22:54:53.915138 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:53.915143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:53.915337 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:53.915511 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:53.915772 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.916550 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.918014 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.918600 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.918766 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:53.918899 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:53.919039 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 22:54:53.919128 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":104,"threads_started":1}
I20260925 22:54:53.919265 28978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:53.913582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:36495 (local address 127.0.0.1:35088)
0925 22:54:53.914905 (+ 1323us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:53.914918 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:53.915005 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:53.915503 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:53.915506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:53.915523 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:53.915619 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:53.915625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.916735 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.916739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:53.917895 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:53.917904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.918732 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.918736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:53.918839 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:53.918872 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 22:54:53.919009 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 22:54:53.919121 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1266,"mutex_wait_us":39,"thread_start_us":94,"threads_started":1}
I20260925 22:54:53.919622 28975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:53.914207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:45563 (local address 127.0.0.1:56242)
0925 22:54:53.915189 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:53.915201 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:53.915277 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:53.915737 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:53.915743 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:53.915760 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:53.915838 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:53.915843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.917162 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.917168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:53.918010 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:53.918021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.919237 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.919241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:53.919331 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:53.919361 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 22:54:53.919444 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 22:54:53.919507 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"thread_start_us":80,"threads_started":1}
I20260925 22:54:53.919737 28980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:53.914590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56242 (local address 127.20.114.193:45563)
0925 22:54:53.914844 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:53.914855 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:53.915307 (+ 452us) server_negotiation.cc:407] Connection header received
0925 22:54:53.915408 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:53.915413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:53.915594 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:53.915738 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:53.915962 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.917052 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.918135 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.918915 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.919054 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:53.919177 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:53.919493 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 22:54:53.919594 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":83,"threads_started":1}
I20260925 22:54:53.921012 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:53.913477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:42959 (local address 127.0.0.1:33760)
0925 22:54:53.914233 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:53.914284 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:53.914402 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:53.915154 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:53.915160 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:53.915185 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:53.915312 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:53.915320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.916744 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.916747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:53.918980 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:53.918992 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.920601 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.920605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:53.920692 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:53.920715 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:54:53.920805 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 22:54:53.920865 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":25,"thread_start_us":83,"threads_started":1}
I20260925 22:54:53.921041 28977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:53.913596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33760 (local address 127.20.114.194:42959)
0925 22:54:53.914690 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:53.914702 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:53.914725 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:54:53.914794 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:53.914803 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:53.915006 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:53.915155 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:53.915437 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.916615 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.919105 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:53.920439 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:53.920615 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:53.920724 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:53.920824 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 22:54:53.920919 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":980,"thread_start_us":120,"threads_started":1}
I20260925 22:54:53.922287 28607 tablet_service.cc:1467] Processing CreateTablet for tablet cb036c0bce654f0988e008d4d97d2ea7 (DEFAULT_TABLE table=test-table [id=edb1c86f33bf48b6a8e2395b111980d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:53.922304 28887 tablet_service.cc:1467] Processing CreateTablet for tablet cb036c0bce654f0988e008d4d97d2ea7 (DEFAULT_TABLE table=test-table [id=edb1c86f33bf48b6a8e2395b111980d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:53.923137 28607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb036c0bce654f0988e008d4d97d2ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:53.923264 28887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb036c0bce654f0988e008d4d97d2ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:53.923491 28747 tablet_service.cc:1467] Processing CreateTablet for tablet cb036c0bce654f0988e008d4d97d2ea7 (DEFAULT_TABLE table=test-table [id=edb1c86f33bf48b6a8e2395b111980d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:53.924338 28747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb036c0bce654f0988e008d4d97d2ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:53.930230 28981 tablet_bootstrap.cc:492] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b: Bootstrap starting.
I20260925 22:54:53.931718 28983 tablet_bootstrap.cc:492] T cb036c0bce654f0988e008d4d97d2ea7 P cef6754891984e688cd7df6a8fc4f3a2: Bootstrap starting.
I20260925 22:54:53.932236 28982 tablet_bootstrap.cc:492] T cb036c0bce654f0988e008d4d97d2ea7 P 3b5b7752b58b4603b857eb4acbae8ef2: Bootstrap starting.
I20260925 22:54:53.932337 28981 tablet_bootstrap.cc:654] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:53.933087 28981 log.cc:824] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:53.933699 28983 tablet_bootstrap.cc:654] T cb036c0bce654f0988e008d4d97d2ea7 P cef6754891984e688cd7df6a8fc4f3a2: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:53.934151 28982 tablet_bootstrap.cc:654] T cb036c0bce654f0988e008d4d97d2ea7 P 3b5b7752b58b4603b857eb4acbae8ef2: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:53.934461 28983 log.cc:824] T cb036c0bce654f0988e008d4d97d2ea7 P cef6754891984e688cd7df6a8fc4f3a2: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:53.934914 28982 log.cc:824] T cb036c0bce654f0988e008d4d97d2ea7 P 3b5b7752b58b4603b857eb4acbae8ef2: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:53.936336 28983 tablet_bootstrap.cc:492] T cb036c0bce654f0988e008d4d97d2ea7 P cef6754891984e688cd7df6a8fc4f3a2: No bootstrap required, opened a new log
I20260925 22:54:53.936519 28983 ts_tablet_manager.cc:1397] T cb036c0bce654f0988e008d4d97d2ea7 P cef6754891984e688cd7df6a8fc4f3a2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 22:54:53.936679 28982 tablet_bootstrap.cc:492] T cb036c0bce654f0988e008d4d97d2ea7 P 3b5b7752b58b4603b857eb4acbae8ef2: No bootstrap required, opened a new log
I20260925 22:54:53.936851 28982 ts_tablet_manager.cc:1397] T cb036c0bce654f0988e008d4d97d2ea7 P 3b5b7752b58b4603b857eb4acbae8ef2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 22:54:53.940382 28983 raft_consensus.cc:348] T cb036c0bce654f0988e008d4d97d2ea7 P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } } peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } }
I20260925 22:54:53.940472 28982 raft_consensus.cc:348] T cb036c0bce654f0988e008d4d97d2ea7 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } } peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } }
I20260925 22:54:53.940653 28982 raft_consensus.cc:374] T cb036c0bce654f0988e008d4d97d2ea7 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:53.940653 28983 raft_consensus.cc:374] T cb036c0bce654f0988e008d4d97d2ea7 P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:53.940711 28983 raft_consensus.cc:729] T cb036c0bce654f0988e008d4d97d2ea7 P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cef6754891984e688cd7df6a8fc4f3a2, State: Initialized, Role: FOLLOWER
I20260925 22:54:53.940711 28982 raft_consensus.cc:729] T cb036c0bce654f0988e008d4d97d2ea7 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5b7752b58b4603b857eb4acbae8ef2, State: Initialized, Role: FOLLOWER
I20260925 22:54:53.941174 28983 consensus_queue.cc:260] T cb036c0bce654f0988e008d4d97d2ea7 P cef6754891984e688cd7df6a8fc4f3a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } } peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } }
I20260925 22:54:53.941191 28982 consensus_queue.cc:260] T cb036c0bce654f0988e008d4d97d2ea7 P 3b5b7752b58b4603b857eb4acbae8ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } } peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } }
I20260925 22:54:53.942019 28953 heartbeater.cc:503] Master 127.20.114.254:41055 was elected leader, sending a full tablet report...
I20260925 22:54:53.942137 28983 ts_tablet_manager.cc:1428] T cb036c0bce654f0988e008d4d97d2ea7 P cef6754891984e688cd7df6a8fc4f3a2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:54:53.942137 28982 ts_tablet_manager.cc:1428] T cb036c0bce654f0988e008d4d97d2ea7 P 3b5b7752b58b4603b857eb4acbae8ef2: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260925 22:54:53.942219 28981 tablet_bootstrap.cc:492] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b: No bootstrap required, opened a new log
I20260925 22:54:53.942389 28981 ts_tablet_manager.cc:1397] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20260925 22:54:53.942651 28813 heartbeater.cc:503] Master 127.20.114.254:41055 was elected leader, sending a full tablet report...
I20260925 22:54:53.946640 28981 raft_consensus.cc:348] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } } peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } }
I20260925 22:54:53.946879 28981 raft_consensus.cc:374] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:53.946915 28981 raft_consensus.cc:729] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfa243493eb4df8bb1907dadb1ae03b, State: Initialized, Role: FOLLOWER
I20260925 22:54:53.947257 28981 consensus_queue.cc:260] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } } peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } }
I20260925 22:54:53.947767 28673 heartbeater.cc:503] Master 127.20.114.254:41055 was elected leader, sending a full tablet report...
I20260925 22:54:53.948112 28981 ts_tablet_manager.cc:1428] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
W20260925 22:54:54.104888 28954 tablet.cc:2369] T cb036c0bce654f0988e008d4d97d2ea7 P 3b5b7752b58b4603b857eb4acbae8ef2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:54:54.122138 28989 raft_consensus.cc:482] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:54:54.122335 28989 raft_consensus.cc:504] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } } peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } }
I20260925 22:54:54.123500 28989 leader_election.cc:288] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cef6754891984e688cd7df6a8fc4f3a2 (127.20.114.194:42959), 3b5b7752b58b4603b857eb4acbae8ef2 (127.20.114.195:36495)
I20260925 22:54:54.127744 28977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:54.123864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:56067 (local address 127.20.114.194:42959)
0925 22:54:54.124053 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:54.124060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:54.124155 (+ 95us) server_negotiation.cc:407] Connection header received
0925 22:54:54.124323 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:54.124326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:54.124377 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:54.124470 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:54.124729 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.125452 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.126672 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.127163 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.127279 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:54.127324 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:54.127529 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 22:54:54.127601 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260925 22:54:54.127786 28990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:54.123757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:42959 (local address 127.20.114.193:56067)
0925 22:54:54.124051 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:54.124066 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:54.124216 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:54.124479 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:54.124482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:54.124499 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:54.124620 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:54.124626 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.125557 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.125563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:54.126472 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:54.126482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.127277 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.127281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:54.127392 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:54.127415 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:54:54.127495 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 22:54:54.127586 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260925 22:54:54.127799 28979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:54.123918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:45277 (local address 127.20.114.195:36495)
0925 22:54:54.124046 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:54.124053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:54.124313 (+ 260us) server_negotiation.cc:407] Connection header received
0925 22:54:54.124464 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:54.124467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:54.124506 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:54.124585 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:54.124857 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.125537 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.126667 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.127100 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.127264 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:54.127317 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:54.127599 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 22:54:54.127695 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:54:54.128430 28991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:54.123820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:36495 (local address 127.20.114.193:45277)
0925 22:54:54.124206 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:54.124217 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:54.124343 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:54.124608 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:54.124611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:54.124626 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:54.124746 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:54.124751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.125640 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.125646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:54.126527 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:54.126536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.127370 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.127374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:54.127465 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:54.127495 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 22:54:54.128230 (+ 735us) client_negotiation.cc:240] Negotiation successful
0925 22:54:54.128303 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":103,"threads_started":1}
I20260925 22:54:54.128515 28767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb036c0bce654f0988e008d4d97d2ea7" candidate_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef6754891984e688cd7df6a8fc4f3a2" is_pre_election: true
I20260925 22:54:54.128880 28767 raft_consensus.cc:2454] T cb036c0bce654f0988e008d4d97d2ea7 P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfa243493eb4df8bb1907dadb1ae03b in term 0.
I20260925 22:54:54.129037 28907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb036c0bce654f0988e008d4d97d2ea7" candidate_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" is_pre_election: true
I20260925 22:54:54.129326 28907 raft_consensus.cc:2454] T cb036c0bce654f0988e008d4d97d2ea7 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfa243493eb4df8bb1907dadb1ae03b in term 0.
I20260925 22:54:54.129390 28558 leader_election.cc:302] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cef6754891984e688cd7df6a8fc4f3a2, fdfa243493eb4df8bb1907dadb1ae03b; no voters:
I20260925 22:54:54.132952 28989 raft_consensus.cc:2781] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:54:54.133054 28989 raft_consensus.cc:482] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:54:54.133117 28989 raft_consensus.cc:3037] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:54.134143 28989 raft_consensus.cc:504] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } } peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } }
W20260925 22:54:54.134449 28814 tablet.cc:2369] T cb036c0bce654f0988e008d4d97d2ea7 P cef6754891984e688cd7df6a8fc4f3a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:54:54.134557 28989 leader_election.cc:288] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b [CANDIDATE]: Term 1 election: Requested vote from peers cef6754891984e688cd7df6a8fc4f3a2 (127.20.114.194:42959), 3b5b7752b58b4603b857eb4acbae8ef2 (127.20.114.195:36495)
I20260925 22:54:54.134829 28767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb036c0bce654f0988e008d4d97d2ea7" candidate_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef6754891984e688cd7df6a8fc4f3a2"
I20260925 22:54:54.134943 28767 raft_consensus.cc:3037] T cb036c0bce654f0988e008d4d97d2ea7 P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:54.134964 28907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb036c0bce654f0988e008d4d97d2ea7" candidate_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5b7752b58b4603b857eb4acbae8ef2"
I20260925 22:54:54.135118 28907 raft_consensus.cc:3037] T cb036c0bce654f0988e008d4d97d2ea7 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:54.136142 28767 raft_consensus.cc:2454] T cb036c0bce654f0988e008d4d97d2ea7 P cef6754891984e688cd7df6a8fc4f3a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfa243493eb4df8bb1907dadb1ae03b in term 1.
I20260925 22:54:54.136439 28558 leader_election.cc:302] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cef6754891984e688cd7df6a8fc4f3a2, fdfa243493eb4df8bb1907dadb1ae03b; no voters:
I20260925 22:54:54.136600 28989 raft_consensus.cc:2781] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:54.136791 28989 raft_consensus.cc:686] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b [term 1 LEADER]: Becoming Leader. State: Replica: fdfa243493eb4df8bb1907dadb1ae03b, State: Running, Role: LEADER
I20260925 22:54:54.136917 28907 raft_consensus.cc:2454] T cb036c0bce654f0988e008d4d97d2ea7 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfa243493eb4df8bb1907dadb1ae03b in term 1.
I20260925 22:54:54.137125 28989 consensus_queue.cc:237] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } } peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } }
I20260925 22:54:54.140385 28469 catalog_manager.cc:5495] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfa243493eb4df8bb1907dadb1ae03b (127.20.114.193). New cstate: current_term: 1 leader_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } health_report { overall_health: HEALTHY } } }
I20260925 22:54:54.157251 28980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:54.154047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56256 (local address 127.20.114.193:45563)
0925 22:54:54.154161 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:54.154173 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:54.154186 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:54:54.154258 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:54.154260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:54.154306 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:54.154390 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:54.154622 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.155104 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.155950 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.156150 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.156232 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:54.156442 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:54.156840 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:54.156983 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:54.157032 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 22:54:54.157077 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:54:54.162680 28907 raft_consensus.cc:1261] T cb036c0bce654f0988e008d4d97d2ea7 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 1 FOLLOWER]: Refusing update from remote peer fdfa243493eb4df8bb1907dadb1ae03b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:54.162734 28767 raft_consensus.cc:1261] T cb036c0bce654f0988e008d4d97d2ea7 P cef6754891984e688cd7df6a8fc4f3a2 [term 1 FOLLOWER]: Refusing update from remote peer fdfa243493eb4df8bb1907dadb1ae03b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:54.163375 28992 consensus_queue.cc:1035] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:54.163554 28989 consensus_queue.cc:1035] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:54.171154 28995 mvcc.cc:205] Tried to move back new op lower bound from 7333383758484074496 to 7333383758388387840. Current Snapshot: MvccSnapshot[applied={T|T < 7333383758484074496}]
I20260925 22:54:54.171592 28996 mvcc.cc:205] Tried to move back new op lower bound from 7333383758484074496 to 7333383758388387840. Current Snapshot: MvccSnapshot[applied={T|T < 7333383758484074496}]
I20260925 22:54:54.179868 28470 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:54.180011 28470 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:54.182298 28470 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537: Sending DeleteTablet for 3 replicas of tablet cb036c0bce654f0988e008d4d97d2ea7
I20260925 22:54:54.183022 28887 tablet_service.cc:1514] Processing DeleteTablet for tablet cb036c0bce654f0988e008d4d97d2ea7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:54 UTC) from {username='oryx', principal='oryx/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:35088
I20260925 22:54:54.183208 28747 tablet_service.cc:1514] Processing DeleteTablet for tablet cb036c0bce654f0988e008d4d97d2ea7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:54 UTC) from {username='oryx', principal='oryx/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:33760
I20260925 22:54:54.183220 28607 tablet_service.cc:1514] Processing DeleteTablet for tablet cb036c0bce654f0988e008d4d97d2ea7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:54 UTC) from {username='oryx', principal='oryx/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:56242
I20260925 22:54:54.183681 29005 tablet_replica.cc:330] T cb036c0bce654f0988e008d4d97d2ea7 P cef6754891984e688cd7df6a8fc4f3a2: stopping tablet replica
I20260925 22:54:54.184000 29005 raft_consensus.cc:2229] T cb036c0bce654f0988e008d4d97d2ea7 P cef6754891984e688cd7df6a8fc4f3a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:54.184031 29004 tablet_replica.cc:330] T cb036c0bce654f0988e008d4d97d2ea7 P 3b5b7752b58b4603b857eb4acbae8ef2: stopping tablet replica
I20260925 22:54:54.184137 28470 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 22:54:54.184298 29005 raft_consensus.cc:2258] T cb036c0bce654f0988e008d4d97d2ea7 P cef6754891984e688cd7df6a8fc4f3a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:54.184302 29004 raft_consensus.cc:2229] T cb036c0bce654f0988e008d4d97d2ea7 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 22:54:54.184502 28470 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:54:54.184537 29004 raft_consensus.cc:2258] T cb036c0bce654f0988e008d4d97d2ea7 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:54.185796 29004 ts_tablet_manager.cc:1905] T cb036c0bce654f0988e008d4d97d2ea7 P 3b5b7752b58b4603b857eb4acbae8ef2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:54.185722 29005 ts_tablet_manager.cc:1905] T cb036c0bce654f0988e008d4d97d2ea7 P cef6754891984e688cd7df6a8fc4f3a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:54.188680 29005 ts_tablet_manager.cc:1918] T cb036c0bce654f0988e008d4d97d2ea7 P cef6754891984e688cd7df6a8fc4f3a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:54:54.188791 29005 log.cc:1196] T cb036c0bce654f0988e008d4d97d2ea7 P cef6754891984e688cd7df6a8fc4f3a2: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/cb036c0bce654f0988e008d4d97d2ea7
I20260925 22:54:54.189103 29004 ts_tablet_manager.cc:1918] T cb036c0bce654f0988e008d4d97d2ea7 P 3b5b7752b58b4603b857eb4acbae8ef2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:54:54.189126 29005 ts_tablet_manager.cc:1939] T cb036c0bce654f0988e008d4d97d2ea7 P cef6754891984e688cd7df6a8fc4f3a2: Deleting consensus metadata
I20260925 22:54:54.189193 29004 log.cc:1196] T cb036c0bce654f0988e008d4d97d2ea7 P 3b5b7752b58b4603b857eb4acbae8ef2: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/cb036c0bce654f0988e008d4d97d2ea7
I20260925 22:54:54.189503 29004 ts_tablet_manager.cc:1939] T cb036c0bce654f0988e008d4d97d2ea7 P 3b5b7752b58b4603b857eb4acbae8ef2: Deleting consensus metadata
I20260925 22:54:54.189790 29006 tablet_replica.cc:330] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b: stopping tablet replica
I20260925 22:54:54.190053 29006 raft_consensus.cc:2229] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:54:54.190379 28747 tablet_service.cc:1467] Processing CreateTablet for tablet ccdec36ee6cd411fbee036d62c09c54c (DEFAULT_TABLE table=test-table [id=33b4724f655446e8a87eb443336b9548]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:54.190455 29006 raft_consensus.cc:2258] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:54.190619 28747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccdec36ee6cd411fbee036d62c09c54c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:54.190622 28457 catalog_manager.cc:4841] TS 3b5b7752b58b4603b857eb4acbae8ef2 (127.20.114.195:36495): tablet cb036c0bce654f0988e008d4d97d2ea7 (table test-table [id=edb1c86f33bf48b6a8e2395b111980d1]) successfully deleted
I20260925 22:54:54.191541 28454 catalog_manager.cc:4841] TS cef6754891984e688cd7df6a8fc4f3a2 (127.20.114.194:42959): tablet cb036c0bce654f0988e008d4d97d2ea7 (table test-table [id=edb1c86f33bf48b6a8e2395b111980d1]) successfully deleted
I20260925 22:54:54.191843 29006 ts_tablet_manager.cc:1905] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:54.192022 28887 tablet_service.cc:1467] Processing CreateTablet for tablet ccdec36ee6cd411fbee036d62c09c54c (DEFAULT_TABLE table=test-table [id=33b4724f655446e8a87eb443336b9548]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:54.192253 28887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccdec36ee6cd411fbee036d62c09c54c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:54.192623 28983 tablet_bootstrap.cc:492] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2: Bootstrap starting.
I20260925 22:54:54.193387 28983 tablet_bootstrap.cc:654] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:54.194312 29006 ts_tablet_manager.cc:1918] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:54:54.194365 28982 tablet_bootstrap.cc:492] T ccdec36ee6cd411fbee036d62c09c54c P 3b5b7752b58b4603b857eb4acbae8ef2: Bootstrap starting.
I20260925 22:54:54.194398 29006 log.cc:1196] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-0/wal/wals/cb036c0bce654f0988e008d4d97d2ea7
I20260925 22:54:54.194721 29006 ts_tablet_manager.cc:1939] T cb036c0bce654f0988e008d4d97d2ea7 P fdfa243493eb4df8bb1907dadb1ae03b: Deleting consensus metadata
I20260925 22:54:54.194813 28607 tablet_service.cc:1467] Processing CreateTablet for tablet ccdec36ee6cd411fbee036d62c09c54c (DEFAULT_TABLE table=test-table [id=33b4724f655446e8a87eb443336b9548]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:54.195073 28607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccdec36ee6cd411fbee036d62c09c54c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:54.195187 28982 tablet_bootstrap.cc:654] T ccdec36ee6cd411fbee036d62c09c54c P 3b5b7752b58b4603b857eb4acbae8ef2: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:54.196218 28983 tablet_bootstrap.cc:492] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2: No bootstrap required, opened a new log
I20260925 22:54:54.196297 28982 tablet_bootstrap.cc:492] T ccdec36ee6cd411fbee036d62c09c54c P 3b5b7752b58b4603b857eb4acbae8ef2: No bootstrap required, opened a new log
I20260925 22:54:54.196303 28983 ts_tablet_manager.cc:1397] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 22:54:54.196372 28982 ts_tablet_manager.cc:1397] T ccdec36ee6cd411fbee036d62c09c54c P 3b5b7752b58b4603b857eb4acbae8ef2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:54:54.196731 28983 raft_consensus.cc:348] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.196771 28982 raft_consensus.cc:348] T ccdec36ee6cd411fbee036d62c09c54c P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.196846 28983 raft_consensus.cc:374] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:54.196884 28982 raft_consensus.cc:374] T ccdec36ee6cd411fbee036d62c09c54c P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:54.196856 28454 catalog_manager.cc:4841] TS fdfa243493eb4df8bb1907dadb1ae03b (127.20.114.193:45563): tablet cb036c0bce654f0988e008d4d97d2ea7 (table test-table [id=edb1c86f33bf48b6a8e2395b111980d1]) successfully deleted
I20260925 22:54:54.196907 28983 raft_consensus.cc:729] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cef6754891984e688cd7df6a8fc4f3a2, State: Initialized, Role: FOLLOWER
I20260925 22:54:54.196926 28982 raft_consensus.cc:729] T ccdec36ee6cd411fbee036d62c09c54c P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5b7752b58b4603b857eb4acbae8ef2, State: Initialized, Role: FOLLOWER
I20260925 22:54:54.197033 28983 consensus_queue.cc:260] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.197033 28982 consensus_queue.cc:260] T ccdec36ee6cd411fbee036d62c09c54c P 3b5b7752b58b4603b857eb4acbae8ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.197283 28982 ts_tablet_manager.cc:1428] T ccdec36ee6cd411fbee036d62c09c54c P 3b5b7752b58b4603b857eb4acbae8ef2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:54.197283 28983 ts_tablet_manager.cc:1428] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:54.197427 28981 tablet_bootstrap.cc:492] T ccdec36ee6cd411fbee036d62c09c54c P fdfa243493eb4df8bb1907dadb1ae03b: Bootstrap starting.
I20260925 22:54:54.198342 28981 tablet_bootstrap.cc:654] T ccdec36ee6cd411fbee036d62c09c54c P fdfa243493eb4df8bb1907dadb1ae03b: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:54.199450 28981 tablet_bootstrap.cc:492] T ccdec36ee6cd411fbee036d62c09c54c P fdfa243493eb4df8bb1907dadb1ae03b: No bootstrap required, opened a new log
I20260925 22:54:54.199530 28981 ts_tablet_manager.cc:1397] T ccdec36ee6cd411fbee036d62c09c54c P fdfa243493eb4df8bb1907dadb1ae03b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260925 22:54:54.199968 28981 raft_consensus.cc:348] T ccdec36ee6cd411fbee036d62c09c54c P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.200073 28981 raft_consensus.cc:374] T ccdec36ee6cd411fbee036d62c09c54c P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:54.200102 28981 raft_consensus.cc:729] T ccdec36ee6cd411fbee036d62c09c54c P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfa243493eb4df8bb1907dadb1ae03b, State: Initialized, Role: FOLLOWER
I20260925 22:54:54.200210 28981 consensus_queue.cc:260] T ccdec36ee6cd411fbee036d62c09c54c P fdfa243493eb4df8bb1907dadb1ae03b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.200434 28981 ts_tablet_manager.cc:1428] T ccdec36ee6cd411fbee036d62c09c54c P fdfa243493eb4df8bb1907dadb1ae03b: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
W20260925 22:54:54.425396 28674 tablet.cc:2369] T ccdec36ee6cd411fbee036d62c09c54c P fdfa243493eb4df8bb1907dadb1ae03b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:54:54.512655 28987 raft_consensus.cc:482] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:54:54.512802 28987 raft_consensus.cc:504] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.514014 28987 leader_election.cc:288] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfa243493eb4df8bb1907dadb1ae03b (127.20.114.193:45563), 3b5b7752b58b4603b857eb4acbae8ef2 (127.20.114.195:36495)
I20260925 22:54:54.517951 28979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:54.514383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:59615 (local address 127.20.114.195:36495)
0925 22:54:54.514557 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:54.514561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:54.514825 (+ 264us) server_negotiation.cc:407] Connection header received
0925 22:54:54.514939 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:54.514942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:54.514987 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:54.515064 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:54.515336 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.515842 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.516947 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.517352 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.517445 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:54.517521 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:54.517758 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 22:54:54.517826 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 22:54:54.517958 28980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:54.514300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:49797 (local address 127.20.114.193:45563)
0925 22:54:54.514540 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:54.514548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:54.514570 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:54:54.514747 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:54.514751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:54.514819 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:54.514899 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:54.515239 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.515774 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.517009 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.517418 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.517537 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:54.517592 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:54.517791 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 22:54:54.517855 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260925 22:54:54.517980 29010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:54.514221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:45563 (local address 127.20.114.194:49797)
0925 22:54:54.514484 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:54.514500 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:54.514623 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:54.514932 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:54.514935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:54.514953 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:54.515104 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:54.515111 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.515889 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.515893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:54.516832 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:54.516842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.517551 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.517555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:54.517650 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:54.517666 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 22:54:54.517758 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 22:54:54.517826 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":81,"thread_start_us":100,"threads_started":1}
I20260925 22:54:54.517985 29011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:54.514313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:36495 (local address 127.20.114.194:59615)
0925 22:54:54.514724 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:54.514738 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:54.514833 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:54.515084 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:54.515086 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:54.515100 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:54.515223 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:54.515230 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.515953 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.515956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:54.516817 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:54.516826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.517473 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.517479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:54.517617 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:54.517642 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:54:54.517716 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 22:54:54.517808 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":47,"thread_start_us":92,"threads_started":1}
I20260925 22:54:54.518565 28627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdec36ee6cd411fbee036d62c09c54c" candidate_uuid: "cef6754891984e688cd7df6a8fc4f3a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" is_pre_election: true
I20260925 22:54:54.518647 28907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdec36ee6cd411fbee036d62c09c54c" candidate_uuid: "cef6754891984e688cd7df6a8fc4f3a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" is_pre_election: true
I20260925 22:54:54.518798 28907 raft_consensus.cc:2454] T ccdec36ee6cd411fbee036d62c09c54c P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cef6754891984e688cd7df6a8fc4f3a2 in term 0.
I20260925 22:54:54.518834 28627 raft_consensus.cc:2454] T ccdec36ee6cd411fbee036d62c09c54c P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cef6754891984e688cd7df6a8fc4f3a2 in term 0.
I20260925 22:54:54.519209 28701 leader_election.cc:302] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3b5b7752b58b4603b857eb4acbae8ef2, cef6754891984e688cd7df6a8fc4f3a2, fdfa243493eb4df8bb1907dadb1ae03b; no voters:
I20260925 22:54:54.519445 28987 raft_consensus.cc:2781] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:54:54.519502 28987 raft_consensus.cc:482] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:54:54.519539 28987 raft_consensus.cc:3037] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:54.520483 28987 raft_consensus.cc:504] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.520795 28987 leader_election.cc:288] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2 [CANDIDATE]: Term 1 election: Requested vote from peers fdfa243493eb4df8bb1907dadb1ae03b (127.20.114.193:45563), 3b5b7752b58b4603b857eb4acbae8ef2 (127.20.114.195:36495)
I20260925 22:54:54.521100 28627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdec36ee6cd411fbee036d62c09c54c" candidate_uuid: "cef6754891984e688cd7df6a8fc4f3a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfa243493eb4df8bb1907dadb1ae03b"
I20260925 22:54:54.521159 28907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdec36ee6cd411fbee036d62c09c54c" candidate_uuid: "cef6754891984e688cd7df6a8fc4f3a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5b7752b58b4603b857eb4acbae8ef2"
I20260925 22:54:54.521268 28627 raft_consensus.cc:3037] T ccdec36ee6cd411fbee036d62c09c54c P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:54.521282 28907 raft_consensus.cc:3037] T ccdec36ee6cd411fbee036d62c09c54c P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:54.522142 28627 raft_consensus.cc:2454] T ccdec36ee6cd411fbee036d62c09c54c P fdfa243493eb4df8bb1907dadb1ae03b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cef6754891984e688cd7df6a8fc4f3a2 in term 1.
I20260925 22:54:54.522156 28907 raft_consensus.cc:2454] T ccdec36ee6cd411fbee036d62c09c54c P 3b5b7752b58b4603b857eb4acbae8ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cef6754891984e688cd7df6a8fc4f3a2 in term 1.
I20260925 22:54:54.522442 28698 leader_election.cc:302] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cef6754891984e688cd7df6a8fc4f3a2, fdfa243493eb4df8bb1907dadb1ae03b; no voters:
I20260925 22:54:54.522609 28987 raft_consensus.cc:2781] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:54.522830 28987 raft_consensus.cc:686] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2 [term 1 LEADER]: Becoming Leader. State: Replica: cef6754891984e688cd7df6a8fc4f3a2, State: Running, Role: LEADER
I20260925 22:54:54.523072 28987 consensus_queue.cc:237] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.525131 28468 catalog_manager.cc:5495] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to cef6754891984e688cd7df6a8fc4f3a2 (127.20.114.194). New cstate: current_term: 1 leader_uuid: "cef6754891984e688cd7df6a8fc4f3a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } health_report { overall_health: UNKNOWN } } }
Sep 25 22:54:54 dist-test-slave-x1lh krb5kdc[28425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376893, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.254@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM
I20260925 22:54:54.587110 29020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:54.578684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44452 (local address 127.20.114.254:41055)
0925 22:54:54.578927 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:54.578937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:54.579033 (+ 96us) server_negotiation.cc:407] Connection header received
0925 22:54:54.579201 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:54.579204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:54.579254 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:54.579351 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:54.580012 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.580553 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.581415 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.581674 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.581792 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:54.584050 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:54.584053 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:54.584056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:54.584082 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:54.585422 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:54.585870 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:54.585875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:54.585877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:54.585953 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:54.586208 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:54.586212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:54.586215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:54.586387 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:54.586490 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:54.586776 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 22:54:54.586935 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":197,"thread_start_us":108,"threads_started":1}
I20260925 22:54:54.587111 29019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:54.578603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:41055 (local address 127.0.0.1:44452)
0925 22:54:54.578925 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:54.578940 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:54.579095 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:54.579369 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:54.579372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:54.579808 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:54.579893 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:54.579900 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.580659 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.580666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:54.581295 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:54.581303 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.581796 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.581800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:54.581894 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:54.582515 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:54.582527 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:54.583915 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:54.585533 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:54.585544 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:54.585550 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:54.585773 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:54.586047 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:54.586050 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:54.586052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:54.586125 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:54.586499 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:54.586507 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:54.586615 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 22:54:54.586731 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 22:54:54.586883 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":5,"thread_start_us":114,"threads_started":1}
I20260925 22:54:54.591077 28468 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:44452:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 22:54:54.591882 28468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:54:54.596397 28607 tablet_service.cc:1467] Processing CreateTablet for tablet 084f79f965184a549955d18741e1d582 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5a7124cf31e9469db7abe32bd20b0de0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:54:54.596451 28747 tablet_service.cc:1467] Processing CreateTablet for tablet 084f79f965184a549955d18741e1d582 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5a7124cf31e9469db7abe32bd20b0de0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:54:54.596596 28887 tablet_service.cc:1467] Processing CreateTablet for tablet 084f79f965184a549955d18741e1d582 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5a7124cf31e9469db7abe32bd20b0de0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:54:54.596813 28607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 084f79f965184a549955d18741e1d582. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:54.596853 28747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 084f79f965184a549955d18741e1d582. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:54.596936 28887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 084f79f965184a549955d18741e1d582. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:54.599395 28983 tablet_bootstrap.cc:492] T 084f79f965184a549955d18741e1d582 P cef6754891984e688cd7df6a8fc4f3a2: Bootstrap starting.
I20260925 22:54:54.599660 28982 tablet_bootstrap.cc:492] T 084f79f965184a549955d18741e1d582 P 3b5b7752b58b4603b857eb4acbae8ef2: Bootstrap starting.
I20260925 22:54:54.600396 28981 tablet_bootstrap.cc:492] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b: Bootstrap starting.
I20260925 22:54:54.600440 28983 tablet_bootstrap.cc:654] T 084f79f965184a549955d18741e1d582 P cef6754891984e688cd7df6a8fc4f3a2: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:54.600591 28982 tablet_bootstrap.cc:654] T 084f79f965184a549955d18741e1d582 P 3b5b7752b58b4603b857eb4acbae8ef2: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:54.601313 28981 tablet_bootstrap.cc:654] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:54.601724 28983 tablet_bootstrap.cc:492] T 084f79f965184a549955d18741e1d582 P cef6754891984e688cd7df6a8fc4f3a2: No bootstrap required, opened a new log
I20260925 22:54:54.601814 28983 ts_tablet_manager.cc:1397] T 084f79f965184a549955d18741e1d582 P cef6754891984e688cd7df6a8fc4f3a2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:54.601872 28982 tablet_bootstrap.cc:492] T 084f79f965184a549955d18741e1d582 P 3b5b7752b58b4603b857eb4acbae8ef2: No bootstrap required, opened a new log
I20260925 22:54:54.601948 28982 ts_tablet_manager.cc:1397] T 084f79f965184a549955d18741e1d582 P 3b5b7752b58b4603b857eb4acbae8ef2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260925 22:54:54.602252 28983 raft_consensus.cc:348] T 084f79f965184a549955d18741e1d582 P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.602378 28983 raft_consensus.cc:374] T 084f79f965184a549955d18741e1d582 P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:54.602327 28982 raft_consensus.cc:348] T 084f79f965184a549955d18741e1d582 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.602411 28983 raft_consensus.cc:729] T 084f79f965184a549955d18741e1d582 P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cef6754891984e688cd7df6a8fc4f3a2, State: Initialized, Role: FOLLOWER
I20260925 22:54:54.602423 28982 raft_consensus.cc:374] T 084f79f965184a549955d18741e1d582 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:54.602459 28982 raft_consensus.cc:729] T 084f79f965184a549955d18741e1d582 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5b7752b58b4603b857eb4acbae8ef2, State: Initialized, Role: FOLLOWER
I20260925 22:54:54.602530 28981 tablet_bootstrap.cc:492] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b: No bootstrap required, opened a new log
I20260925 22:54:54.602607 28981 ts_tablet_manager.cc:1397] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:54.602576 28982 consensus_queue.cc:260] T 084f79f965184a549955d18741e1d582 P 3b5b7752b58b4603b857eb4acbae8ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.602610 28983 consensus_queue.cc:260] T 084f79f965184a549955d18741e1d582 P cef6754891984e688cd7df6a8fc4f3a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.602874 28982 ts_tablet_manager.cc:1428] T 084f79f965184a549955d18741e1d582 P 3b5b7752b58b4603b857eb4acbae8ef2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:54:54.602824 28987 tablet_replica.cc:439] T 084f79f965184a549955d18741e1d582 P cef6754891984e688cd7df6a8fc4f3a2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } } }
I20260925 22:54:54.602846 28988 tablet_replica.cc:439] T 084f79f965184a549955d18741e1d582 P 3b5b7752b58b4603b857eb4acbae8ef2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } } }
I20260925 22:54:54.602942 28987 tablet_replica.cc:442] T 084f79f965184a549955d18741e1d582 P cef6754891984e688cd7df6a8fc4f3a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:54:54.602978 28988 tablet_replica.cc:442] T 084f79f965184a549955d18741e1d582 P 3b5b7752b58b4603b857eb4acbae8ef2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:54:54.603019 28981 raft_consensus.cc:348] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.603112 28981 raft_consensus.cc:374] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:54.603148 28981 raft_consensus.cc:729] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfa243493eb4df8bb1907dadb1ae03b, State: Initialized, Role: FOLLOWER
I20260925 22:54:54.603242 28981 consensus_queue.cc:260] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.603487 28981 ts_tablet_manager.cc:1428] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:54.603435 28992 tablet_replica.cc:439] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } } }
I20260925 22:54:54.603544 28992 tablet_replica.cc:442] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:54:54.603551 28983 ts_tablet_manager.cc:1428] T 084f79f965184a549955d18741e1d582 P cef6754891984e688cd7df6a8fc4f3a2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:54:54.621712 28992 raft_consensus.cc:482] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:54:54.621865 28992 raft_consensus.cc:504] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.622287 28992 leader_election.cc:288] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cef6754891984e688cd7df6a8fc4f3a2 (127.20.114.194:42959), 3b5b7752b58b4603b857eb4acbae8ef2 (127.20.114.195:36495)
I20260925 22:54:54.622637 28767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084f79f965184a549955d18741e1d582" candidate_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef6754891984e688cd7df6a8fc4f3a2" is_pre_election: true
I20260925 22:54:54.622659 28907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084f79f965184a549955d18741e1d582" candidate_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" is_pre_election: true
I20260925 22:54:54.622802 28767 raft_consensus.cc:2454] T 084f79f965184a549955d18741e1d582 P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfa243493eb4df8bb1907dadb1ae03b in term 0.
I20260925 22:54:54.622802 28907 raft_consensus.cc:2454] T 084f79f965184a549955d18741e1d582 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfa243493eb4df8bb1907dadb1ae03b in term 0.
I20260925 22:54:54.623118 28558 leader_election.cc:302] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3b5b7752b58b4603b857eb4acbae8ef2, cef6754891984e688cd7df6a8fc4f3a2, fdfa243493eb4df8bb1907dadb1ae03b; no voters:
I20260925 22:54:54.623282 28992 raft_consensus.cc:2781] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:54:54.623344 28992 raft_consensus.cc:482] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:54:54.623394 28992 raft_consensus.cc:3037] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:54.624370 28992 raft_consensus.cc:504] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.624706 28992 leader_election.cc:288] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b [CANDIDATE]: Term 1 election: Requested vote from peers cef6754891984e688cd7df6a8fc4f3a2 (127.20.114.194:42959), 3b5b7752b58b4603b857eb4acbae8ef2 (127.20.114.195:36495)
I20260925 22:54:54.625065 28767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084f79f965184a549955d18741e1d582" candidate_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef6754891984e688cd7df6a8fc4f3a2"
I20260925 22:54:54.625167 28767 raft_consensus.cc:3037] T 084f79f965184a549955d18741e1d582 P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:54.625118 28907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084f79f965184a549955d18741e1d582" candidate_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5b7752b58b4603b857eb4acbae8ef2"
I20260925 22:54:54.625270 28907 raft_consensus.cc:3037] T 084f79f965184a549955d18741e1d582 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:54.626363 28767 raft_consensus.cc:2454] T 084f79f965184a549955d18741e1d582 P cef6754891984e688cd7df6a8fc4f3a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfa243493eb4df8bb1907dadb1ae03b in term 1.
I20260925 22:54:54.626396 28907 raft_consensus.cc:2454] T 084f79f965184a549955d18741e1d582 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfa243493eb4df8bb1907dadb1ae03b in term 1.
I20260925 22:54:54.626668 28561 leader_election.cc:302] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5b7752b58b4603b857eb4acbae8ef2, fdfa243493eb4df8bb1907dadb1ae03b; no voters:
I20260925 22:54:54.626863 28992 raft_consensus.cc:2781] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:54.626950 28992 raft_consensus.cc:686] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b [term 1 LEADER]: Becoming Leader. State: Replica: fdfa243493eb4df8bb1907dadb1ae03b, State: Running, Role: LEADER
I20260925 22:54:54.627100 28992 consensus_queue.cc:237] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.627830 28989 tablet_replica.cc:439] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b: TxnStatusTablet state changed. Reason: New leader fdfa243493eb4df8bb1907dadb1ae03b. Latest consensus state: current_term: 1 leader_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } } }
I20260925 22:54:54.627969 28989 tablet_replica.cc:442] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:54:54.628262 29025 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 22:54:54.628502 29025 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 22:54:54.628928 28469 catalog_manager.cc:5495] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfa243493eb4df8bb1907dadb1ae03b (127.20.114.193). New cstate: current_term: 1 leader_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } health_report { overall_health: UNKNOWN } } }
I20260925 22:54:54.645792 28980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:54.641625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38392 (local address 127.20.114.193:45563)
0925 22:54:54.642074 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:54.642078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:54.642090 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:54:54.642132 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:54.642135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:54.642179 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:54.642258 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:54.642536 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.643322 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.644500 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.644842 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.644958 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:54.645325 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:54.645412 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:54.645482 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:54.645609 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 22:54:54.645663 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375}
I20260925 22:54:54.645802 29019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:54.641522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:45563 (local address 127.0.0.1:38392)
0925 22:54:54.641776 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:54.641800 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:54.641901 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:54.642303 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:54.642306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:54.642315 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:54:54.642424 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:54.642430 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.643429 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.643433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:54.644382 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:54.644392 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.645036 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.645040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:54.645132 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:54.645181 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:54:54.645503 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:54:54.645516 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 22:54:54.645583 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 22:54:54.645670 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260925 22:54:54.648341 28767 raft_consensus.cc:1261] T 084f79f965184a549955d18741e1d582 P cef6754891984e688cd7df6a8fc4f3a2 [term 1 FOLLOWER]: Refusing update from remote peer fdfa243493eb4df8bb1907dadb1ae03b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:54.648365 28907 raft_consensus.cc:1261] T 084f79f965184a549955d18741e1d582 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 1 FOLLOWER]: Refusing update from remote peer fdfa243493eb4df8bb1907dadb1ae03b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:54.648759 28992 consensus_queue.cc:1035] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:54.649152 28992 consensus_queue.cc:1035] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:54.650162 28987 tablet_replica.cc:439] T 084f79f965184a549955d18741e1d582 P cef6754891984e688cd7df6a8fc4f3a2: TxnStatusTablet state changed. Reason: New leader fdfa243493eb4df8bb1907dadb1ae03b. Latest consensus state: current_term: 1 leader_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } } }
I20260925 22:54:54.650274 28987 tablet_replica.cc:442] T 084f79f965184a549955d18741e1d582 P cef6754891984e688cd7df6a8fc4f3a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:54:54.650628 28988 tablet_replica.cc:439] T 084f79f965184a549955d18741e1d582 P 3b5b7752b58b4603b857eb4acbae8ef2: TxnStatusTablet state changed. Reason: New leader fdfa243493eb4df8bb1907dadb1ae03b. Latest consensus state: current_term: 1 leader_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } } }
I20260925 22:54:54.650745 28988 tablet_replica.cc:442] T 084f79f965184a549955d18741e1d582 P 3b5b7752b58b4603b857eb4acbae8ef2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:54:54.652318 28992 tablet_replica.cc:439] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } } }
I20260925 22:54:54.652426 28992 tablet_replica.cc:442] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:54:54.654351 28989 tablet_replica.cc:439] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } } }
I20260925 22:54:54.654445 28989 tablet_replica.cc:442] T 084f79f965184a549955d18741e1d582 P fdfa243493eb4df8bb1907dadb1ae03b: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:54:54.655795 28997 mvcc.cc:205] Tried to move back new op lower bound from 7333383760475332608 to 7333383760394108928. Current Snapshot: MvccSnapshot[applied={T|T < 7333383760475332608}]
I20260925 22:54:54.655967 28987 tablet_replica.cc:439] T 084f79f965184a549955d18741e1d582 P cef6754891984e688cd7df6a8fc4f3a2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } } }
I20260925 22:54:54.656064 28987 tablet_replica.cc:442] T 084f79f965184a549955d18741e1d582 P cef6754891984e688cd7df6a8fc4f3a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:54:54.656719 28988 tablet_replica.cc:439] T 084f79f965184a549955d18741e1d582 P 3b5b7752b58b4603b857eb4acbae8ef2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } } }
I20260925 22:54:54.656838 28988 tablet_replica.cc:442] T 084f79f965184a549955d18741e1d582 P 3b5b7752b58b4603b857eb4acbae8ef2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:54:54.665521 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:54.659030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37110 (local address 127.20.114.194:42959)
0925 22:54:54.659292 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:54.659300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:54.661181 (+ 1881us) server_negotiation.cc:407] Connection header received
0925 22:54:54.661408 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:54.661412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:54.661487 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:54.661606 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:54.661979 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.662840 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.663794 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.664096 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.664219 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:54.664498 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:54.665038 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:54.665163 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:54.665209 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 22:54:54.665260 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":98,"threads_started":1}
I20260925 22:54:54.675675 28980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:54.672274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:58711 (local address 127.20.114.193:45563)
0925 22:54:54.672400 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:54.672403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:54.672527 (+ 124us) server_negotiation.cc:407] Connection header received
0925 22:54:54.672632 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:54.672634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:54.672677 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:54.672738 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:54.673014 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.673487 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.674725 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.674925 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.674996 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:54.675301 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:54.675362 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:54.675412 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:54.675524 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 22:54:54.675572 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:54:54.675745 29035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:54.672139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:45563 (local address 127.20.114.194:58711)
0925 22:54:54.672438 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:54.672453 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:54.672553 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:54.672769 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:54.672772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:54.672779 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:54:54.672931 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:54.672937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.673584 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.673587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:54.674612 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:54.674622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.675023 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.675027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:54.675124 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:54.675173 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:54:54.675439 (+ 266us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:54:54.675445 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:54:54.675509 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 22:54:54.675584 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":93,"threads_started":1}
I20260925 22:54:54.682602 28907 raft_consensus.cc:1261] T ccdec36ee6cd411fbee036d62c09c54c P 3b5b7752b58b4603b857eb4acbae8ef2 [term 1 FOLLOWER]: Refusing update from remote peer cef6754891984e688cd7df6a8fc4f3a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:54.682907 28627 raft_consensus.cc:1261] T ccdec36ee6cd411fbee036d62c09c54c P fdfa243493eb4df8bb1907dadb1ae03b [term 1 FOLLOWER]: Refusing update from remote peer cef6754891984e688cd7df6a8fc4f3a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:54.683065 28987 consensus_queue.cc:1035] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:54.683359 29012 consensus_queue.cc:1035] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:54.705979 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:54.702181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:57763 (local address 127.20.114.194:42959)
0925 22:54:54.702294 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:54.702299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:54.702471 (+ 172us) server_negotiation.cc:407] Connection header received
0925 22:54:54.702615 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:54.702619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:54.702669 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:54.702740 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:54.702999 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.703507 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.704608 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.704967 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.705047 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:54.705524 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:54.705629 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:54.705721 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:54.705837 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 22:54:54.705880 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 22:54:54.706064 29041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:54.702124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:42959 (local address 127.20.114.193:57763)
0925 22:54:54.702390 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:54.702402 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:54.702518 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:54.702761 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:54.702764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:54.702772 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:54:54.702901 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:54.702906 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.703614 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.703619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:54.704477 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:54.704487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.705165 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.705169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:54.705291 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:54.705359 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:54:54.705741 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:54:54.705748 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 22:54:54.705830 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 22:54:54.705908 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":73,"threads_started":1}
I20260925 22:54:54.729645 28468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:54.729817 28468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:54.731707 28468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537: Sending DeleteTablet for 3 replicas of tablet ccdec36ee6cd411fbee036d62c09c54c
I20260925 22:54:54.732378 28887 tablet_service.cc:1514] Processing DeleteTablet for tablet ccdec36ee6cd411fbee036d62c09c54c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:54 UTC) from {username='oryx', principal='oryx/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:35088
I20260925 22:54:54.732381 28607 tablet_service.cc:1514] Processing DeleteTablet for tablet ccdec36ee6cd411fbee036d62c09c54c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:54 UTC) from {username='oryx', principal='oryx/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:56242
I20260925 22:54:54.733060 28747 tablet_service.cc:1514] Processing DeleteTablet for tablet ccdec36ee6cd411fbee036d62c09c54c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:54 UTC) from {username='oryx', principal='oryx/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:33760
I20260925 22:54:54.733111 29045 tablet_replica.cc:330] T ccdec36ee6cd411fbee036d62c09c54c P fdfa243493eb4df8bb1907dadb1ae03b: stopping tablet replica
I20260925 22:54:54.733243 29045 raft_consensus.cc:2229] T ccdec36ee6cd411fbee036d62c09c54c P fdfa243493eb4df8bb1907dadb1ae03b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:54.733351 29045 raft_consensus.cc:2258] T ccdec36ee6cd411fbee036d62c09c54c P fdfa243493eb4df8bb1907dadb1ae03b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:54.733551 29044 tablet_replica.cc:330] T ccdec36ee6cd411fbee036d62c09c54c P 3b5b7752b58b4603b857eb4acbae8ef2: stopping tablet replica
I20260925 22:54:54.733678 29044 raft_consensus.cc:2229] T ccdec36ee6cd411fbee036d62c09c54c P 3b5b7752b58b4603b857eb4acbae8ef2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:54.733788 29044 raft_consensus.cc:2258] T ccdec36ee6cd411fbee036d62c09c54c P 3b5b7752b58b4603b857eb4acbae8ef2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:54.737795 29045 ts_tablet_manager.cc:1905] T ccdec36ee6cd411fbee036d62c09c54c P fdfa243493eb4df8bb1907dadb1ae03b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:54.740154 29045 ts_tablet_manager.cc:1918] T ccdec36ee6cd411fbee036d62c09c54c P fdfa243493eb4df8bb1907dadb1ae03b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:54:54.740247 29045 log.cc:1196] T ccdec36ee6cd411fbee036d62c09c54c P fdfa243493eb4df8bb1907dadb1ae03b: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-0/wal/wals/ccdec36ee6cd411fbee036d62c09c54c
I20260925 22:54:54.740513 29045 ts_tablet_manager.cc:1939] T ccdec36ee6cd411fbee036d62c09c54c P fdfa243493eb4df8bb1907dadb1ae03b: Deleting consensus metadata
I20260925 22:54:54.740913 29044 ts_tablet_manager.cc:1905] T ccdec36ee6cd411fbee036d62c09c54c P 3b5b7752b58b4603b857eb4acbae8ef2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:54.741011 29046 tablet_replica.cc:330] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2: stopping tablet replica
I20260925 22:54:54.741145 29046 raft_consensus.cc:2229] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:54:54.741279 28454 catalog_manager.cc:4841] TS fdfa243493eb4df8bb1907dadb1ae03b (127.20.114.193:45563): tablet ccdec36ee6cd411fbee036d62c09c54c (table test-table [id=33b4724f655446e8a87eb443336b9548]) successfully deleted
I20260925 22:54:54.741461 29046 raft_consensus.cc:2258] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:54.743314 29044 ts_tablet_manager.cc:1918] T ccdec36ee6cd411fbee036d62c09c54c P 3b5b7752b58b4603b857eb4acbae8ef2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:54:54.743412 29044 log.cc:1196] T ccdec36ee6cd411fbee036d62c09c54c P 3b5b7752b58b4603b857eb4acbae8ef2: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/ccdec36ee6cd411fbee036d62c09c54c
I20260925 22:54:54.743649 29044 ts_tablet_manager.cc:1939] T ccdec36ee6cd411fbee036d62c09c54c P 3b5b7752b58b4603b857eb4acbae8ef2: Deleting consensus metadata
I20260925 22:54:54.744303 28457 catalog_manager.cc:4841] TS 3b5b7752b58b4603b857eb4acbae8ef2 (127.20.114.195:36495): tablet ccdec36ee6cd411fbee036d62c09c54c (table test-table [id=33b4724f655446e8a87eb443336b9548]) successfully deleted
I20260925 22:54:54.745687 29046 ts_tablet_manager.cc:1905] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:54.748091 29046 ts_tablet_manager.cc:1918] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:54:54.748196 29046 log.cc:1196] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/ccdec36ee6cd411fbee036d62c09c54c
I20260925 22:54:54.748441 29046 ts_tablet_manager.cc:1939] T ccdec36ee6cd411fbee036d62c09c54c P cef6754891984e688cd7df6a8fc4f3a2: Deleting consensus metadata
I20260925 22:54:54.749094 28454 catalog_manager.cc:4841] TS cef6754891984e688cd7df6a8fc4f3a2 (127.20.114.194:42959): tablet ccdec36ee6cd411fbee036d62c09c54c (table test-table [id=33b4724f655446e8a87eb443336b9548]) successfully deleted
Sep 25 22:54:54 dist-test-slave-x1lh krb5kdc[28425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:54:54 dist-test-slave-x1lh krb5kdc[28425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:54:54 dist-test-slave-x1lh krb5kdc[28425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:54:54.768083 29020 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:54.761239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44466 (local address 127.20.114.254:41055)
0925 22:54:54.761375 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:54.761380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:54.761872 (+ 492us) server_negotiation.cc:407] Connection header received
0925 22:54:54.762028 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:54.762031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:54.762120 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:54.762228 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:54.763013 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.763558 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.764430 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.764670 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.764783 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:54.767957 (+ 3174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44466: BlockingRecv error: recv got EOF from 127.0.0.1:44466 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 22:54:54 dist-test-slave-x1lh krb5kdc[28425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM
I20260925 22:54:54.781713 29020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:54.773272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.20.114.254:41055)
0925 22:54:54.773383 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:54.773387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:54.773622 (+ 235us) server_negotiation.cc:407] Connection header received
0925 22:54:54.773742 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:54.773745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:54.773792 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:54.773879 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:54.774467 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.774969 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.775758 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:54.775972 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:54.776080 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:54.778413 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:54.778417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:54.778419 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:54.778450 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:54.779859 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:54.780283 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:54.780288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:54.780290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:54.780372 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:54.780645 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:54.780650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:54.780652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:54.780829 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:54.781029 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:54.781401 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 22:54:54.781518 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:54:54.784272 28468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:54:54.784742 28468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:54:54.789369 28607 tablet_service.cc:1467] Processing CreateTablet for tablet a8fad2132cc742a69010fdab6475b122 (DEFAULT_TABLE table=test-table [id=08b8b000c1c3496cb52b441208ea0d6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:54.789705 28607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8fad2132cc742a69010fdab6475b122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:54.791976 28981 tablet_bootstrap.cc:492] T a8fad2132cc742a69010fdab6475b122 P fdfa243493eb4df8bb1907dadb1ae03b: Bootstrap starting.
I20260925 22:54:54.792364 28887 tablet_service.cc:1467] Processing CreateTablet for tablet a8fad2132cc742a69010fdab6475b122 (DEFAULT_TABLE table=test-table [id=08b8b000c1c3496cb52b441208ea0d6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:54.792623 28887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8fad2132cc742a69010fdab6475b122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:54.793074 28981 tablet_bootstrap.cc:654] T a8fad2132cc742a69010fdab6475b122 P fdfa243493eb4df8bb1907dadb1ae03b: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:54.794888 28982 tablet_bootstrap.cc:492] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2: Bootstrap starting.
I20260925 22:54:54.795429 28747 tablet_service.cc:1467] Processing CreateTablet for tablet a8fad2132cc742a69010fdab6475b122 (DEFAULT_TABLE table=test-table [id=08b8b000c1c3496cb52b441208ea0d6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:54.795707 28747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8fad2132cc742a69010fdab6475b122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:54.795784 28982 tablet_bootstrap.cc:654] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:54.797022 28982 tablet_bootstrap.cc:492] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2: No bootstrap required, opened a new log
I20260925 22:54:54.797103 28982 ts_tablet_manager.cc:1397] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260925 22:54:54.797602 28982 raft_consensus.cc:348] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.797712 28982 raft_consensus.cc:374] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:54.797745 28982 raft_consensus.cc:729] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5b7752b58b4603b857eb4acbae8ef2, State: Initialized, Role: FOLLOWER
I20260925 22:54:54.797874 28982 consensus_queue.cc:260] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.798161 28982 ts_tablet_manager.cc:1428] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:54.799036 28983 tablet_bootstrap.cc:492] T a8fad2132cc742a69010fdab6475b122 P cef6754891984e688cd7df6a8fc4f3a2: Bootstrap starting.
I20260925 22:54:54.799953 28983 tablet_bootstrap.cc:654] T a8fad2132cc742a69010fdab6475b122 P cef6754891984e688cd7df6a8fc4f3a2: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:54.800486 28981 tablet_bootstrap.cc:492] T a8fad2132cc742a69010fdab6475b122 P fdfa243493eb4df8bb1907dadb1ae03b: No bootstrap required, opened a new log
I20260925 22:54:54.800585 28981 ts_tablet_manager.cc:1397] T a8fad2132cc742a69010fdab6475b122 P fdfa243493eb4df8bb1907dadb1ae03b: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20260925 22:54:54.801179 28981 raft_consensus.cc:348] T a8fad2132cc742a69010fdab6475b122 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.801329 28981 raft_consensus.cc:374] T a8fad2132cc742a69010fdab6475b122 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:54.801369 28981 raft_consensus.cc:729] T a8fad2132cc742a69010fdab6475b122 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfa243493eb4df8bb1907dadb1ae03b, State: Initialized, Role: FOLLOWER
I20260925 22:54:54.801527 28981 consensus_queue.cc:260] T a8fad2132cc742a69010fdab6475b122 P fdfa243493eb4df8bb1907dadb1ae03b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.801783 28981 ts_tablet_manager.cc:1428] T a8fad2132cc742a69010fdab6475b122 P fdfa243493eb4df8bb1907dadb1ae03b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:54.805651 28983 tablet_bootstrap.cc:492] T a8fad2132cc742a69010fdab6475b122 P cef6754891984e688cd7df6a8fc4f3a2: No bootstrap required, opened a new log
I20260925 22:54:54.805758 28983 ts_tablet_manager.cc:1397] T a8fad2132cc742a69010fdab6475b122 P cef6754891984e688cd7df6a8fc4f3a2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260925 22:54:54.806170 28983 raft_consensus.cc:348] T a8fad2132cc742a69010fdab6475b122 P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.806290 28983 raft_consensus.cc:374] T a8fad2132cc742a69010fdab6475b122 P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:54.806330 28983 raft_consensus.cc:729] T a8fad2132cc742a69010fdab6475b122 P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cef6754891984e688cd7df6a8fc4f3a2, State: Initialized, Role: FOLLOWER
I20260925 22:54:54.806458 28983 consensus_queue.cc:260] T a8fad2132cc742a69010fdab6475b122 P cef6754891984e688cd7df6a8fc4f3a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:54.806701 28983 ts_tablet_manager.cc:1428] T a8fad2132cc742a69010fdab6475b122 P cef6754891984e688cd7df6a8fc4f3a2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:55.023262 28988 raft_consensus.cc:482] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:54:55.023402 28988 raft_consensus.cc:504] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:55.024485 28988 leader_election.cc:288] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfa243493eb4df8bb1907dadb1ae03b (127.20.114.193:45563), cef6754891984e688cd7df6a8fc4f3a2 (127.20.114.194:42959)
I20260925 22:54:55.028636 28980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:55.024822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:53455 (local address 127.20.114.193:45563)
0925 22:54:55.024971 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:55.024975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:55.025058 (+ 83us) server_negotiation.cc:407] Connection header received
0925 22:54:55.025239 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:55.025242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:55.025295 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:55.025389 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:55.025757 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:55.026441 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:55.027629 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:55.028066 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:55.028158 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:55.028245 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:55.028441 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 22:54:55.028511 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:54:55.028685 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:55.024693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:45563 (local address 127.20.114.195:53455)
0925 22:54:55.024954 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:55.024970 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:55.025119 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:55.025416 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:55.025419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:55.025441 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:55.025609 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:55.025616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:55.026557 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:55.026560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:55.027503 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:55.027512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:55.028175 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:55.028179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:55.028302 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:55.028327 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:54:55.028418 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 22:54:55.028489 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":1,"spinlock_wait_cycles":21376,"thread_start_us":96,"threads_started":1}
I20260925 22:54:55.028885 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:55.025317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:60843 (local address 127.20.114.194:42959)
0925 22:54:55.025440 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:55.025445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:55.025458 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:54:55.025624 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:55.025627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:55.025681 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:55.025786 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:54:55.025998 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:55.026644 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:55.027751 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:55.028196 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:55.028313 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:55.028388 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:55.028643 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 22:54:55.028739 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:54:55.028970 29063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:55.025179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:42959 (local address 127.20.114.195:60843)
0925 22:54:55.025396 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:55.025411 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:55.025507 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:55.025783 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:55.025786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:55.025803 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:54:55.025886 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:55.025891 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:55.026760 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:55.026763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:55.027555 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:55.027588 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:55.028444 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:55.028448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:55.028524 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:55.028544 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:54:55.028615 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 22:54:55.028673 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":59,"threads_started":1}
I20260925 22:54:55.029268 28627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fad2132cc742a69010fdab6475b122" candidate_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" is_pre_election: true
I20260925 22:54:55.029428 28627 raft_consensus.cc:2454] T a8fad2132cc742a69010fdab6475b122 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5b7752b58b4603b857eb4acbae8ef2 in term 0.
I20260925 22:54:55.029672 28767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fad2132cc742a69010fdab6475b122" candidate_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef6754891984e688cd7df6a8fc4f3a2" is_pre_election: true
I20260925 22:54:55.029810 28767 raft_consensus.cc:2454] T a8fad2132cc742a69010fdab6475b122 P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5b7752b58b4603b857eb4acbae8ef2 in term 0.
I20260925 22:54:55.029803 28838 leader_election.cc:302] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5b7752b58b4603b857eb4acbae8ef2, fdfa243493eb4df8bb1907dadb1ae03b; no voters:
I20260925 22:54:55.030074 28988 raft_consensus.cc:2781] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:54:55.030138 28988 raft_consensus.cc:482] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:54:55.030177 28988 raft_consensus.cc:3037] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:55.031141 28988 raft_consensus.cc:504] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:55.031486 28988 leader_election.cc:288] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2 [CANDIDATE]: Term 1 election: Requested vote from peers fdfa243493eb4df8bb1907dadb1ae03b (127.20.114.193:45563), cef6754891984e688cd7df6a8fc4f3a2 (127.20.114.194:42959)
I20260925 22:54:55.031817 28627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fad2132cc742a69010fdab6475b122" candidate_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfa243493eb4df8bb1907dadb1ae03b"
I20260925 22:54:55.031881 28767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fad2132cc742a69010fdab6475b122" candidate_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef6754891984e688cd7df6a8fc4f3a2"
I20260925 22:54:55.031929 28627 raft_consensus.cc:3037] T a8fad2132cc742a69010fdab6475b122 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:55.032029 28767 raft_consensus.cc:3037] T a8fad2132cc742a69010fdab6475b122 P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:55.032915 28627 raft_consensus.cc:2454] T a8fad2132cc742a69010fdab6475b122 P fdfa243493eb4df8bb1907dadb1ae03b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5b7752b58b4603b857eb4acbae8ef2 in term 1.
I20260925 22:54:55.033093 28767 raft_consensus.cc:2454] T a8fad2132cc742a69010fdab6475b122 P cef6754891984e688cd7df6a8fc4f3a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5b7752b58b4603b857eb4acbae8ef2 in term 1.
I20260925 22:54:55.033371 28838 leader_election.cc:302] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5b7752b58b4603b857eb4acbae8ef2, fdfa243493eb4df8bb1907dadb1ae03b; no voters:
I20260925 22:54:55.033636 28988 raft_consensus.cc:2781] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:55.033847 28988 raft_consensus.cc:686] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5b7752b58b4603b857eb4acbae8ef2, State: Running, Role: LEADER
I20260925 22:54:55.034096 28988 consensus_queue.cc:237] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:55.036278 28468 catalog_manager.cc:5495] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5b7752b58b4603b857eb4acbae8ef2 (127.20.114.195). New cstate: current_term: 1 leader_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } health_report { overall_health: HEALTHY } } }
I20260925 22:54:55.097926 29067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:55.094309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52304 (local address 127.20.114.195:36495)
0925 22:54:55.094635 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:55.094641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:55.094655 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:54:55.094741 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:55.094744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:55.094811 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:55.094912 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:55.095137 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:55.095658 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:55.096439 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:55.096659 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:55.096740 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:55.097041 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:55.097510 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:55.097643 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:55.097701 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 22:54:55.097754 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":140,"threads_started":1}
I20260925 22:54:55.100431 28767 raft_consensus.cc:1261] T a8fad2132cc742a69010fdab6475b122 P cef6754891984e688cd7df6a8fc4f3a2 [term 1 FOLLOWER]: Refusing update from remote peer 3b5b7752b58b4603b857eb4acbae8ef2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:55.100431 28627 raft_consensus.cc:1261] T a8fad2132cc742a69010fdab6475b122 P fdfa243493eb4df8bb1907dadb1ae03b [term 1 FOLLOWER]: Refusing update from remote peer 3b5b7752b58b4603b857eb4acbae8ef2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:55.100963 29064 consensus_queue.cc:1035] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:55.101197 28988 consensus_queue.cc:1035] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:55.111454 28468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:55.111586 28468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:55.113694 28468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537: Sending DeleteTablet for 3 replicas of tablet a8fad2132cc742a69010fdab6475b122
I20260925 22:54:55.114270 28747 tablet_service.cc:1514] Processing DeleteTablet for tablet a8fad2132cc742a69010fdab6475b122 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:55 UTC) from {username='oryx', principal='oryx/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:33760
I20260925 22:54:55.114295 28887 tablet_service.cc:1514] Processing DeleteTablet for tablet a8fad2132cc742a69010fdab6475b122 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:55 UTC) from {username='oryx', principal='oryx/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:35088
I20260925 22:54:55.114462 28607 tablet_service.cc:1514] Processing DeleteTablet for tablet a8fad2132cc742a69010fdab6475b122 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:55 UTC) from {username='oryx', principal='oryx/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:56242
I20260925 22:54:55.114492 29044 tablet_replica.cc:330] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2: stopping tablet replica
I20260925 22:54:55.114643 29046 tablet_replica.cc:330] T a8fad2132cc742a69010fdab6475b122 P cef6754891984e688cd7df6a8fc4f3a2: stopping tablet replica
I20260925 22:54:55.114656 29044 raft_consensus.cc:2229] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:54:55.114799 29046 raft_consensus.cc:2229] T a8fad2132cc742a69010fdab6475b122 P cef6754891984e688cd7df6a8fc4f3a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:55.114929 29046 raft_consensus.cc:2258] T a8fad2132cc742a69010fdab6475b122 P cef6754891984e688cd7df6a8fc4f3a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:55.114961 29044 raft_consensus.cc:2258] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:55.114956 28468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 22:54:55.115096 29045 tablet_replica.cc:330] T a8fad2132cc742a69010fdab6475b122 P fdfa243493eb4df8bb1907dadb1ae03b: stopping tablet replica
I20260925 22:54:55.115198 29045 raft_consensus.cc:2229] T a8fad2132cc742a69010fdab6475b122 P fdfa243493eb4df8bb1907dadb1ae03b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:55.115320 29045 raft_consensus.cc:2258] T a8fad2132cc742a69010fdab6475b122 P fdfa243493eb4df8bb1907dadb1ae03b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 22:54:55.115365 28468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:54:55.115775 29044 ts_tablet_manager.cc:1905] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:55.116047 29045 ts_tablet_manager.cc:1905] T a8fad2132cc742a69010fdab6475b122 P fdfa243493eb4df8bb1907dadb1ae03b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:55.118309 29044 ts_tablet_manager.cc:1918] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:54:55.118393 29044 log.cc:1196] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/a8fad2132cc742a69010fdab6475b122
I20260925 22:54:55.118489 29045 ts_tablet_manager.cc:1918] T a8fad2132cc742a69010fdab6475b122 P fdfa243493eb4df8bb1907dadb1ae03b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:54:55.118552 29045 log.cc:1196] T a8fad2132cc742a69010fdab6475b122 P fdfa243493eb4df8bb1907dadb1ae03b: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-0/wal/wals/a8fad2132cc742a69010fdab6475b122
I20260925 22:54:55.118675 29044 ts_tablet_manager.cc:1939] T a8fad2132cc742a69010fdab6475b122 P 3b5b7752b58b4603b857eb4acbae8ef2: Deleting consensus metadata
I20260925 22:54:55.118798 29045 ts_tablet_manager.cc:1939] T a8fad2132cc742a69010fdab6475b122 P fdfa243493eb4df8bb1907dadb1ae03b: Deleting consensus metadata
I20260925 22:54:55.119298 28457 catalog_manager.cc:4841] TS 3b5b7752b58b4603b857eb4acbae8ef2 (127.20.114.195:36495): tablet a8fad2132cc742a69010fdab6475b122 (table test-table [id=08b8b000c1c3496cb52b441208ea0d6e]) successfully deleted
I20260925 22:54:55.120186 28607 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8efbc5e31640d7a786dd9d7ab0a949 (DEFAULT_TABLE table=test-table [id=c534ce451bc1453badafac84bf9f8be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:55.120211 28747 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8efbc5e31640d7a786dd9d7ab0a949 (DEFAULT_TABLE table=test-table [id=c534ce451bc1453badafac84bf9f8be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:55.120431 28607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8efbc5e31640d7a786dd9d7ab0a949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:55.120465 28747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8efbc5e31640d7a786dd9d7ab0a949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:55.122481 28981 tablet_bootstrap.cc:492] T 6c8efbc5e31640d7a786dd9d7ab0a949 P fdfa243493eb4df8bb1907dadb1ae03b: Bootstrap starting.
I20260925 22:54:55.122676 28454 catalog_manager.cc:4841] TS fdfa243493eb4df8bb1907dadb1ae03b (127.20.114.193:45563): tablet a8fad2132cc742a69010fdab6475b122 (table test-table [id=08b8b000c1c3496cb52b441208ea0d6e]) successfully deleted
I20260925 22:54:55.122880 28983 tablet_bootstrap.cc:492] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2: Bootstrap starting.
I20260925 22:54:55.123368 28981 tablet_bootstrap.cc:654] T 6c8efbc5e31640d7a786dd9d7ab0a949 P fdfa243493eb4df8bb1907dadb1ae03b: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:55.123467 28887 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8efbc5e31640d7a786dd9d7ab0a949 (DEFAULT_TABLE table=test-table [id=c534ce451bc1453badafac84bf9f8be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:54:55.123718 28983 tablet_bootstrap.cc:654] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:55.123724 28887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8efbc5e31640d7a786dd9d7ab0a949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:55.125298 28983 tablet_bootstrap.cc:492] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2: No bootstrap required, opened a new log
I20260925 22:54:55.125365 29046 ts_tablet_manager.cc:1905] T a8fad2132cc742a69010fdab6475b122 P cef6754891984e688cd7df6a8fc4f3a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:55.125375 28983 ts_tablet_manager.cc:1397] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 22:54:55.125828 28983 raft_consensus.cc:348] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:55.125931 28983 raft_consensus.cc:374] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:55.125962 28983 raft_consensus.cc:729] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cef6754891984e688cd7df6a8fc4f3a2, State: Initialized, Role: FOLLOWER
I20260925 22:54:55.126039 28982 tablet_bootstrap.cc:492] T 6c8efbc5e31640d7a786dd9d7ab0a949 P 3b5b7752b58b4603b857eb4acbae8ef2: Bootstrap starting.
I20260925 22:54:55.126067 28983 consensus_queue.cc:260] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:55.126320 28983 ts_tablet_manager.cc:1428] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:55.127003 28982 tablet_bootstrap.cc:654] T 6c8efbc5e31640d7a786dd9d7ab0a949 P 3b5b7752b58b4603b857eb4acbae8ef2: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:55.127646 29046 ts_tablet_manager.cc:1918] T a8fad2132cc742a69010fdab6475b122 P cef6754891984e688cd7df6a8fc4f3a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:54:55.127712 29046 log.cc:1196] T a8fad2132cc742a69010fdab6475b122 P cef6754891984e688cd7df6a8fc4f3a2: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/a8fad2132cc742a69010fdab6475b122
I20260925 22:54:55.128563 29046 ts_tablet_manager.cc:1939] T a8fad2132cc742a69010fdab6475b122 P cef6754891984e688cd7df6a8fc4f3a2: Deleting consensus metadata
I20260925 22:54:55.129271 28454 catalog_manager.cc:4841] TS cef6754891984e688cd7df6a8fc4f3a2 (127.20.114.194:42959): tablet a8fad2132cc742a69010fdab6475b122 (table test-table [id=08b8b000c1c3496cb52b441208ea0d6e]) successfully deleted
I20260925 22:54:55.129640 28981 tablet_bootstrap.cc:492] T 6c8efbc5e31640d7a786dd9d7ab0a949 P fdfa243493eb4df8bb1907dadb1ae03b: No bootstrap required, opened a new log
I20260925 22:54:55.129702 28981 ts_tablet_manager.cc:1397] T 6c8efbc5e31640d7a786dd9d7ab0a949 P fdfa243493eb4df8bb1907dadb1ae03b: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260925 22:54:55.129988 28981 raft_consensus.cc:348] T 6c8efbc5e31640d7a786dd9d7ab0a949 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:55.130076 28981 raft_consensus.cc:374] T 6c8efbc5e31640d7a786dd9d7ab0a949 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:55.130097 28981 raft_consensus.cc:729] T 6c8efbc5e31640d7a786dd9d7ab0a949 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfa243493eb4df8bb1907dadb1ae03b, State: Initialized, Role: FOLLOWER
I20260925 22:54:55.130231 28981 consensus_queue.cc:260] T 6c8efbc5e31640d7a786dd9d7ab0a949 P fdfa243493eb4df8bb1907dadb1ae03b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:55.130492 28981 ts_tablet_manager.cc:1428] T 6c8efbc5e31640d7a786dd9d7ab0a949 P fdfa243493eb4df8bb1907dadb1ae03b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:54:55.133566 28982 tablet_bootstrap.cc:492] T 6c8efbc5e31640d7a786dd9d7ab0a949 P 3b5b7752b58b4603b857eb4acbae8ef2: No bootstrap required, opened a new log
I20260925 22:54:55.133668 28982 ts_tablet_manager.cc:1397] T 6c8efbc5e31640d7a786dd9d7ab0a949 P 3b5b7752b58b4603b857eb4acbae8ef2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260925 22:54:55.134066 28982 raft_consensus.cc:348] T 6c8efbc5e31640d7a786dd9d7ab0a949 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:55.134184 28982 raft_consensus.cc:374] T 6c8efbc5e31640d7a786dd9d7ab0a949 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:55.134217 28982 raft_consensus.cc:729] T 6c8efbc5e31640d7a786dd9d7ab0a949 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5b7752b58b4603b857eb4acbae8ef2, State: Initialized, Role: FOLLOWER
I20260925 22:54:55.134337 28982 consensus_queue.cc:260] T 6c8efbc5e31640d7a786dd9d7ab0a949 P 3b5b7752b58b4603b857eb4acbae8ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:55.134593 28982 ts_tablet_manager.cc:1428] T 6c8efbc5e31640d7a786dd9d7ab0a949 P 3b5b7752b58b4603b857eb4acbae8ef2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:54:55.496435 29039 raft_consensus.cc:482] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:54:55.496572 29039 raft_consensus.cc:504] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:55.496981 29039 leader_election.cc:288] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfa243493eb4df8bb1907dadb1ae03b (127.20.114.193:45563), 3b5b7752b58b4603b857eb4acbae8ef2 (127.20.114.195:36495)
I20260925 22:54:55.497421 28627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8efbc5e31640d7a786dd9d7ab0a949" candidate_uuid: "cef6754891984e688cd7df6a8fc4f3a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" is_pre_election: true
I20260925 22:54:55.497444 28907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8efbc5e31640d7a786dd9d7ab0a949" candidate_uuid: "cef6754891984e688cd7df6a8fc4f3a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" is_pre_election: true
I20260925 22:54:55.497555 28627 raft_consensus.cc:2454] T 6c8efbc5e31640d7a786dd9d7ab0a949 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cef6754891984e688cd7df6a8fc4f3a2 in term 0.
I20260925 22:54:55.497614 28907 raft_consensus.cc:2454] T 6c8efbc5e31640d7a786dd9d7ab0a949 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cef6754891984e688cd7df6a8fc4f3a2 in term 0.
I20260925 22:54:55.497840 28698 leader_election.cc:302] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cef6754891984e688cd7df6a8fc4f3a2, fdfa243493eb4df8bb1907dadb1ae03b; no voters:
I20260925 22:54:55.498016 29039 raft_consensus.cc:2781] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:54:55.498075 29039 raft_consensus.cc:482] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:54:55.498114 29039 raft_consensus.cc:3037] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:55.499084 29039 raft_consensus.cc:504] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:55.499406 29039 leader_election.cc:288] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2 [CANDIDATE]: Term 1 election: Requested vote from peers fdfa243493eb4df8bb1907dadb1ae03b (127.20.114.193:45563), 3b5b7752b58b4603b857eb4acbae8ef2 (127.20.114.195:36495)
I20260925 22:54:55.499701 28627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8efbc5e31640d7a786dd9d7ab0a949" candidate_uuid: "cef6754891984e688cd7df6a8fc4f3a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfa243493eb4df8bb1907dadb1ae03b"
I20260925 22:54:55.499835 28627 raft_consensus.cc:3037] T 6c8efbc5e31640d7a786dd9d7ab0a949 P fdfa243493eb4df8bb1907dadb1ae03b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:55.499802 28907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8efbc5e31640d7a786dd9d7ab0a949" candidate_uuid: "cef6754891984e688cd7df6a8fc4f3a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5b7752b58b4603b857eb4acbae8ef2"
I20260925 22:54:55.499936 28907 raft_consensus.cc:3037] T 6c8efbc5e31640d7a786dd9d7ab0a949 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:55.500806 28627 raft_consensus.cc:2454] T 6c8efbc5e31640d7a786dd9d7ab0a949 P fdfa243493eb4df8bb1907dadb1ae03b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cef6754891984e688cd7df6a8fc4f3a2 in term 1.
I20260925 22:54:55.500818 28907 raft_consensus.cc:2454] T 6c8efbc5e31640d7a786dd9d7ab0a949 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cef6754891984e688cd7df6a8fc4f3a2 in term 1.
I20260925 22:54:55.501104 28698 leader_election.cc:302] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cef6754891984e688cd7df6a8fc4f3a2, fdfa243493eb4df8bb1907dadb1ae03b; no voters:
I20260925 22:54:55.501274 29039 raft_consensus.cc:2781] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:55.501462 29039 raft_consensus.cc:686] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2 [term 1 LEADER]: Becoming Leader. State: Replica: cef6754891984e688cd7df6a8fc4f3a2, State: Running, Role: LEADER
I20260925 22:54:55.501626 29039 consensus_queue.cc:237] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } }
I20260925 22:54:55.503229 28469 catalog_manager.cc:5495] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to cef6754891984e688cd7df6a8fc4f3a2 (127.20.114.194). New cstate: current_term: 1 leader_uuid: "cef6754891984e688cd7df6a8fc4f3a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cef6754891984e688cd7df6a8fc4f3a2" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 } health_report { overall_health: UNKNOWN } } }
I20260925 22:54:55.589313 29078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:55.586599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37118 (local address 127.20.114.194:42959)
0925 22:54:55.586820 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:55.586824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:55.586835 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:54:55.586870 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:55.586872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:55.586910 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:55.586979 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:55.587166 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:55.587656 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:55.588410 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:55.588616 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:55.588686 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:55.588959 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:55.589024 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:55.589079 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:55.589159 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 22:54:55.589202 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":92,"threads_started":1}
I20260925 22:54:55.594767 28627 raft_consensus.cc:1261] T 6c8efbc5e31640d7a786dd9d7ab0a949 P fdfa243493eb4df8bb1907dadb1ae03b [term 1 FOLLOWER]: Refusing update from remote peer cef6754891984e688cd7df6a8fc4f3a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:55.594913 28907 raft_consensus.cc:1261] T 6c8efbc5e31640d7a786dd9d7ab0a949 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 1 FOLLOWER]: Refusing update from remote peer cef6754891984e688cd7df6a8fc4f3a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:55.595120 29074 consensus_queue.cc:1035] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfa243493eb4df8bb1907dadb1ae03b" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:55.595263 29039 consensus_queue.cc:1035] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5b7752b58b4603b857eb4acbae8ef2" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 36495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:55.625154 28469 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:55.625320 28469 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:54:55.627149 28469 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de923d24b4aa4c2bad4faf9b72a86537: Sending DeleteTablet for 3 replicas of tablet 6c8efbc5e31640d7a786dd9d7ab0a949
I20260925 22:54:55.627703 28607 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8efbc5e31640d7a786dd9d7ab0a949 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:55 UTC) from {username='oryx', principal='oryx/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:56242
I20260925 22:54:55.627730 28887 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8efbc5e31640d7a786dd9d7ab0a949 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:55 UTC) from {username='oryx', principal='oryx/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:35088
I20260925 22:54:55.627842 28747 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8efbc5e31640d7a786dd9d7ab0a949 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:54:55 UTC) from {username='oryx', principal='oryx/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:33760
I20260925 22:54:55.628084 29085 tablet_replica.cc:330] T 6c8efbc5e31640d7a786dd9d7ab0a949 P 3b5b7752b58b4603b857eb4acbae8ef2: stopping tablet replica
I20260925 22:54:55.628098 29046 tablet_replica.cc:330] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2: stopping tablet replica
I20260925 22:54:55.628140 29084 tablet_replica.cc:330] T 6c8efbc5e31640d7a786dd9d7ab0a949 P fdfa243493eb4df8bb1907dadb1ae03b: stopping tablet replica
I20260925 22:54:55.628198 29046 raft_consensus.cc:2229] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:54:55.628198 29085 raft_consensus.cc:2229] T 6c8efbc5e31640d7a786dd9d7ab0a949 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:55.628281 29084 raft_consensus.cc:2229] T 6c8efbc5e31640d7a786dd9d7ab0a949 P fdfa243493eb4df8bb1907dadb1ae03b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:54:55.628306 29085 raft_consensus.cc:2258] T 6c8efbc5e31640d7a786dd9d7ab0a949 P 3b5b7752b58b4603b857eb4acbae8ef2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:55.628365 29046 raft_consensus.cc:2258] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:55.628373 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 28542
I20260925 22:54:55.628392 29084 raft_consensus.cc:2258] T 6c8efbc5e31640d7a786dd9d7ab0a949 P fdfa243493eb4df8bb1907dadb1ae03b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:54:55.629494 29085 ts_tablet_manager.cc:1905] T 6c8efbc5e31640d7a786dd9d7ab0a949 P 3b5b7752b58b4603b857eb4acbae8ef2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:55.631186 29046 ts_tablet_manager.cc:1905] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:54:55.632081 29085 ts_tablet_manager.cc:1918] T 6c8efbc5e31640d7a786dd9d7ab0a949 P 3b5b7752b58b4603b857eb4acbae8ef2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:54:55.632166 29085 log.cc:1196] T 6c8efbc5e31640d7a786dd9d7ab0a949 P 3b5b7752b58b4603b857eb4acbae8ef2: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/6c8efbc5e31640d7a786dd9d7ab0a949
I20260925 22:54:55.632417 29085 ts_tablet_manager.cc:1939] T 6c8efbc5e31640d7a786dd9d7ab0a949 P 3b5b7752b58b4603b857eb4acbae8ef2: Deleting consensus metadata
I20260925 22:54:55.633508 29046 ts_tablet_manager.cc:1918] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:54:55.633601 29046 log.cc:1196] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/6c8efbc5e31640d7a786dd9d7ab0a949
I20260925 22:54:55.633849 29046 ts_tablet_manager.cc:1939] T 6c8efbc5e31640d7a786dd9d7ab0a949 P cef6754891984e688cd7df6a8fc4f3a2: Deleting consensus metadata
I20260925 22:54:55.634490 28454 catalog_manager.cc:4841] TS cef6754891984e688cd7df6a8fc4f3a2 (127.20.114.194:42959): tablet 6c8efbc5e31640d7a786dd9d7ab0a949 (table test-table [id=c534ce451bc1453badafac84bf9f8be0]) successfully deleted
I20260925 22:54:55.640117 28457 catalog_manager.cc:4841] TS 3b5b7752b58b4603b857eb4acbae8ef2 (127.20.114.195:36495): tablet 6c8efbc5e31640d7a786dd9d7ab0a949 (table test-table [id=c534ce451bc1453badafac84bf9f8be0]) successfully deleted
W20260925 22:54:55.640264 28454 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.114.193:45563 (error 108)
I20260925 22:54:55.640404 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 28682
I20260925 22:54:55.641680 29087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:55.640973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:45563 (local address 127.0.0.1:38398)
0925 22:54:55.641540 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:55.641599 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.193:45563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":51,"threads_started":1}
W20260925 22:54:55.641824 28454 catalog_manager.cc:4584] TS fdfa243493eb4df8bb1907dadb1ae03b (127.20.114.193:45563): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c8efbc5e31640d7a786dd9d7ab0a949: Network error: Client connection negotiation failed: client connection to 127.20.114.193:45563: connect: Connection refused (error 111)
I20260925 22:54:55.647432 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 28822
I20260925 22:54:55.653995 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 28438
2026-09-25T22:54:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3850 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:54:55 dist-test-slave-x1lh krb5kdc[29090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:54:55 dist-test-slave-x1lh krb5kdc[29090](info): set up 2 sockets
Sep 25 22:54:55 dist-test-slave-x1lh krb5kdc[29090](info): commencing operation
krb5kdc: starting...
W20260925 22:54:56.697578 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:54:56 dist-test-slave-x1lh krb5kdc[29090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:54:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:54:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:54:56.816470 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:41033
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:41033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:56.924453 29103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:56.924759 29103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:56.924829 29103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:56.928746 29103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:56.928834 29103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:56.928866 29103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:56.928947 29103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:56.929005 29103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:56.934309 29103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:41033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:41033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:56.935343 29103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:56.936336 29103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:56.944705 29109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:56.944696 29111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:56.944725 29108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:56.945055 29103 server_base.cc:1034] running on GCE node
I20260925 22:54:56.945559 29103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:56.946630 29103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:56.947786 29103 hybrid_clock.cc:648] HybridClock initialized: now 1790376896947768 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T22:54:56Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1309 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:54:57 dist-test-slave-x1lh krb5kdc[29116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:54:57 dist-test-slave-x1lh krb5kdc[29116](info): set up 2 sockets
Sep 25 22:54:57 dist-test-slave-x1lh krb5kdc[29116](info): commencing operation
krb5kdc: starting...
W20260925 22:54:57.993649 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:54:58 dist-test-slave-x1lh krb5kdc[29116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:54:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:54:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:54:58.111742 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:43439
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:42447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:43439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:58.222853 29129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:58.223098 29129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:58.223165 29129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:58.227303 29129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:58.227356 29129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:58.227376 29129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:58.227432 29129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:58.227466 29129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:58.232842 29129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:42447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:43439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:43439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:58.234087 29129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:58.235136 29129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:58.243928 29135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:58.243956 29129 server_base.cc:1034] running on GCE node
W20260925 22:54:58.243963 29134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:58.243973 29137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:58.244648 29129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:58.245800 29129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:58.246979 29129 hybrid_clock.cc:648] HybridClock initialized: now 1790376898246945 us; error 48 us; skew 500 ppm
Sep 25 22:54:58 dist-test-slave-x1lh krb5kdc[29116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376898, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:58.249776 29129 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:54:58.250651 29129 webserver.cc:460] Webserver started at http://127.20.114.254:40391/ using document root <none> and password file <none>
I20260925 22:54:58.251225 29129 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:58.251313 29129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:58.251537 29129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:58.253638 29129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "b08caa5462e14ada8510219d9c89243b"
format_stamp: "Formatted at 2026-09-25 22:54:58 on dist-test-slave-x1lh"
server_key: "9cc7ed331d1d3ae21aaf9b8dd8323234"
server_key_iv: "e5903e73570872e2b84c251f52f3db38"
server_key_version: "encryptionkey@0"
I20260925 22:54:58.254199 29129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "b08caa5462e14ada8510219d9c89243b"
format_stamp: "Formatted at 2026-09-25 22:54:58 on dist-test-slave-x1lh"
server_key: "9cc7ed331d1d3ae21aaf9b8dd8323234"
server_key_iv: "e5903e73570872e2b84c251f52f3db38"
server_key_version: "encryptionkey@0"
I20260925 22:54:58.257872 29129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:54:58.260248 29144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:58.261507 29129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:58.261656 29129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "b08caa5462e14ada8510219d9c89243b"
format_stamp: "Formatted at 2026-09-25 22:54:58 on dist-test-slave-x1lh"
server_key: "9cc7ed331d1d3ae21aaf9b8dd8323234"
server_key_iv: "e5903e73570872e2b84c251f52f3db38"
server_key_version: "encryptionkey@0"
I20260925 22:54:58.261771 29129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:58.278986 29129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:58.281888 29129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:58.282095 29129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:58.289189 29129 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:43439
I20260925 22:54:58.289216 29196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:43439 every 8 connection(s)
I20260925 22:54:58.290138 29129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:54:58.292799 29197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:58.298183 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 29129
I20260925 22:54:58.298278 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:54:58.298302 29197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b: Bootstrap starting.
I20260925 22:54:58.298552 20939 external_mini_cluster.cc:1428] Setting key b6edc719373710c83085b1a7f218181e
I20260925 22:54:58.300832 29197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:58.301573 29197 log.cc:824] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:58.303359 29197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b: No bootstrap required, opened a new log
Sep 25 22:54:58 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:58.307090 29197 raft_consensus.cc:348] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08caa5462e14ada8510219d9c89243b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43439 } }
I20260925 22:54:58.307363 29197 raft_consensus.cc:374] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:58.307428 29197 raft_consensus.cc:729] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b08caa5462e14ada8510219d9c89243b, State: Initialized, Role: FOLLOWER
I20260925 22:54:58.307915 29197 consensus_queue.cc:260] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08caa5462e14ada8510219d9c89243b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43439 } }
I20260925 22:54:58.308041 29197 raft_consensus.cc:388] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:54:58.308130 29197 raft_consensus.cc:482] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:54:58.308241 29197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:58.309291 29197 raft_consensus.cc:504] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08caa5462e14ada8510219d9c89243b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43439 } }
I20260925 22:54:58.309660 29197 leader_election.cc:302] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b08caa5462e14ada8510219d9c89243b; no voters:
I20260925 22:54:58.310006 29197 leader_election.cc:288] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:54:58.310101 29202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:58.310328 29202 raft_consensus.cc:686] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [term 1 LEADER]: Becoming Leader. State: Replica: b08caa5462e14ada8510219d9c89243b, State: Running, Role: LEADER
I20260925 22:54:58.310659 29202 consensus_queue.cc:237] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08caa5462e14ada8510219d9c89243b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43439 } }
I20260925 22:54:58.311309 29197 sys_catalog.cc:564] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:58.312425 29203 sys_catalog.cc:455] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b08caa5462e14ada8510219d9c89243b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08caa5462e14ada8510219d9c89243b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43439 } } }
I20260925 22:54:58.312551 29203 sys_catalog.cc:458] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:58.313110 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:58.299909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39704 (local address 127.20.114.254:43439)
0925 22:54:58.300328 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:58.300343 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:58.300376 (+ 33us) server_negotiation.cc:407] Connection header received
0925 22:54:58.301125 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:58.301150 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:58.301471 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:58.301794 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:58.302585 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:58.303606 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:58.304431 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:58.304960 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:58.305123 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:58.307831 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:58.307836 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:58.307847 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:58.307871 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:58.309687 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:58.310196 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:58.310204 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:58.310213 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:58.310365 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:58.310660 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:58.310663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:58.310664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:58.311132 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:58.311318 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:58.311741 (+ 423us) server_negotiation.cc:299] Negotiation successful
0925 22:54:58.312050 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":130,"threads_started":1}
I20260925 22:54:58.313105 29204 sys_catalog.cc:455] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b08caa5462e14ada8510219d9c89243b. Latest consensus state: current_term: 1 leader_uuid: "b08caa5462e14ada8510219d9c89243b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08caa5462e14ada8510219d9c89243b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43439 } } }
I20260925 22:54:58.313208 29204 sys_catalog.cc:458] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:58.313730 29211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:54:58.316787 29211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:54:58.322293 29211 catalog_manager.cc:1347] Generated new cluster ID: d71332ca992e423c95b92828f9ec9189
I20260925 22:54:58.322356 29211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:58.342216 29211 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:54:58.343441 29211 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:58.352350 29211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b: Generated new TSK 0
I20260925 22:54:58.353166 29211 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.114.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
I20260925 22:54:58.384114 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:43439
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:42447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:58.510811 29223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:58.511091 29223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:58.511191 29223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:58.515437 29223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:58.515509 29223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:58.515621 29223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:54:58.521011 29223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:42447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:43439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:58.522112 29223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:58.523195 29223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:58.532997 29228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:58.532991 29231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:58.533047 29223 server_base.cc:1034] running on GCE node
W20260925 22:54:58.532997 29229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:58.533716 29223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:58.534518 29223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:58.535698 29223 hybrid_clock.cc:648] HybridClock initialized: now 1790376898535680 us; error 35 us; skew 500 ppm
Sep 25 22:54:58 dist-test-slave-x1lh krb5kdc[29116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376898, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:58.538597 29223 init.cc:377] Logged in from keytab as kudu/127.20.114.193@KRBTEST.COM (short username kudu)
I20260925 22:54:58.539425 29223 webserver.cc:460] Webserver started at http://127.20.114.193:34257/ using document root <none> and password file <none>
I20260925 22:54:58.539959 29223 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:58.540016 29223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:58.540179 29223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:58.542266 29223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "e20d03800a2240cab16241b1fe73f18c"
format_stamp: "Formatted at 2026-09-25 22:54:58 on dist-test-slave-x1lh"
server_key: "5201aaa9b49d21b78f754f1757c8cc6f"
server_key_iv: "f60d1ef9721db8d4fb60be6ebad44895"
server_key_version: "encryptionkey@0"
I20260925 22:54:58.542800 29223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "e20d03800a2240cab16241b1fe73f18c"
format_stamp: "Formatted at 2026-09-25 22:54:58 on dist-test-slave-x1lh"
server_key: "5201aaa9b49d21b78f754f1757c8cc6f"
server_key_iv: "f60d1ef9721db8d4fb60be6ebad44895"
server_key_version: "encryptionkey@0"
I20260925 22:54:58.546490 29223 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:54:58.548837 29238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:58.550150 29223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:58.550247 29223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "e20d03800a2240cab16241b1fe73f18c"
format_stamp: "Formatted at 2026-09-25 22:54:58 on dist-test-slave-x1lh"
server_key: "5201aaa9b49d21b78f754f1757c8cc6f"
server_key_iv: "f60d1ef9721db8d4fb60be6ebad44895"
server_key_version: "encryptionkey@0"
I20260925 22:54:58.550333 29223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:58.564567 29223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:58.568040 29223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:58.568193 29223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:58.568735 29223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:58.569628 29223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:58.569677 29223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:58.569717 29223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:58.569746 29223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:58.578604 29223 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:34217
I20260925 22:54:58.578625 29351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:34217 every 8 connection(s)
I20260925 22:54:58.579540 29223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:54:58.581128 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 29223
I20260925 22:54:58.581264 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:54:58.581616 20939 external_mini_cluster.cc:1428] Setting key 782b80839eb70b9da55f653d7de2e645
Sep 25 22:54:58 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376898, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.114.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.194@KRBTEST.COM" created.
I20260925 22:54:58.594515 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:58.581376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:52779 (local address 127.20.114.254:43439)
0925 22:54:58.581521 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:58.581526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:58.582726 (+ 1200us) server_negotiation.cc:407] Connection header received
0925 22:54:58.583629 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:58.583633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:58.583684 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:58.583760 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:58.585304 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:58.585825 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:58.587608 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:58.588028 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:58.588182 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:58.590881 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:58.590887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:58.590891 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:58.590924 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:58.592377 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:58.592988 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:58.592991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:58.592992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:58.593065 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:58.593355 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:58.593357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:58.593358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:58.593523 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:58.593594 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:58.594113 (+ 519us) server_negotiation.cc:299] Negotiation successful
0925 22:54:58.594261 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:54:58.595275 29354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:58.581828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.193:52779)
0925 22:54:58.582569 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:58.582620 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:58.583428 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:58.583924 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:58.583931 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:58.584372 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:58.585132 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:58.585156 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:58.585949 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:58.585955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:58.587490 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:58.587502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:58.588215 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:58.588218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:58.588357 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:58.589026 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:58.589042 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:58.590709 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:58.592478 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:58.592482 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:58.592493 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:58.592895 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:58.593143 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:58.593146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:58.593148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:58.593273 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:58.593616 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:58.593626 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:58.593895 (+ 269us) client_negotiation.cc:769] Sending connection context
0925 22:54:58.594065 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 22:54:58.594394 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":140,"threads_started":1}
I20260925 22:54:58.596586 29352 heartbeater.cc:347] Connected to a master server at 127.20.114.254:43439
I20260925 22:54:58.596837 29352 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:58.597343 29352 heartbeater.cc:511] Master 127.20.114.254:43439 requested a full tablet report, sending...
I20260925 22:54:58.598829 29161 ts_manager.cc:194] Registered new tserver with Master: e20d03800a2240cab16241b1fe73f18c (127.20.114.193:34217)
I20260925 22:54:58.599949 29161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:52779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
I20260925 22:54:58.609889 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:43439
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:42447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:58.719110 29357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:58.719395 29357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:58.719465 29357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:58.723469 29357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:58.723553 29357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:58.723645 29357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:54:58.728686 29357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:42447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:43439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:58.729775 29357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:58.730791 29357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:58.740320 29362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:58.740298 29363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:58.740320 29365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:58.740561 29357 server_base.cc:1034] running on GCE node
I20260925 22:54:58.741113 29357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:58.741757 29357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:58.742933 29357 hybrid_clock.cc:648] HybridClock initialized: now 1790376898742911 us; error 45 us; skew 500 ppm
Sep 25 22:54:58 dist-test-slave-x1lh krb5kdc[29116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376898, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:58.745711 29357 init.cc:377] Logged in from keytab as kudu/127.20.114.194@KRBTEST.COM (short username kudu)
I20260925 22:54:58.746496 29357 webserver.cc:460] Webserver started at http://127.20.114.194:42625/ using document root <none> and password file <none>
I20260925 22:54:58.747035 29357 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:58.747123 29357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:58.747347 29357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:58.749344 29357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "807ca216fd9745aa9a5f2d014e667762"
format_stamp: "Formatted at 2026-09-25 22:54:58 on dist-test-slave-x1lh"
server_key: "b6220768f98836710c98c7e9bebbc2e8"
server_key_iv: "62314798f843857d290e7b6f3783752d"
server_key_version: "encryptionkey@0"
I20260925 22:54:58.749902 29357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "807ca216fd9745aa9a5f2d014e667762"
format_stamp: "Formatted at 2026-09-25 22:54:58 on dist-test-slave-x1lh"
server_key: "b6220768f98836710c98c7e9bebbc2e8"
server_key_iv: "62314798f843857d290e7b6f3783752d"
server_key_version: "encryptionkey@0"
I20260925 22:54:58.753381 29357 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:54:58.755690 29372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:58.756847 29357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:58.757019 29357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "807ca216fd9745aa9a5f2d014e667762"
format_stamp: "Formatted at 2026-09-25 22:54:58 on dist-test-slave-x1lh"
server_key: "b6220768f98836710c98c7e9bebbc2e8"
server_key_iv: "62314798f843857d290e7b6f3783752d"
server_key_version: "encryptionkey@0"
I20260925 22:54:58.757122 29357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:58.777729 29357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:58.781122 29357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:58.781332 29357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:58.781939 29357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:58.782790 29357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:58.782862 29357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:58.782958 29357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:58.783008 29357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:58.791735 29357 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:39777
I20260925 22:54:58.791754 29485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:39777 every 8 connection(s)
I20260925 22:54:58.792619 29357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:54:58.796160 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 29357
I20260925 22:54:58.796304 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:54:58.796581 20939 external_mini_cluster.cc:1428] Setting key 9c082d42d3a21c5b26b2edc39491e8c2
Sep 25 22:54:58 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376898, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.114.195@KRBTEST.COM; defaulting to no policy
I20260925 22:54:58.807019 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:58.794372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:43273 (local address 127.20.114.254:43439)
0925 22:54:58.794521 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:58.794525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:58.795534 (+ 1009us) server_negotiation.cc:407] Connection header received
0925 22:54:58.796505 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:58.796509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:58.796573 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:58.796659 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:58.797960 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:58.798706 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:58.800063 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:58.800282 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:58.800365 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:58.803221 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:58.803227 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:58.803231 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:58.803265 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:58.804955 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:58.805433 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:58.805437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:58.805439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:58.805532 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:58.805848 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:58.805851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:58.805855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:58.806114 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:58.806226 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:58.806739 (+ 513us) server_negotiation.cc:299] Negotiation successful
0925 22:54:58.806880 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:54:58.807982 29488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:58.794757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.194:43273)
0925 22:54:58.795385 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:58.795425 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:58.796209 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:58.796849 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:58.796856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:58.797327 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:58.797799 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:58.797812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:58.798847 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:58.798849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:58.799935 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:58.799949 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:58.800435 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:58.800439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:58.800666 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:58.801396 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:58.801412 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:58.803050 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:58.805066 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:58.805071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:58.805082 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:58.805331 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:58.805626 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:58.805628 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:58.805629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:58.805751 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:58.806228 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:58.806233 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:58.806454 (+ 221us) client_negotiation.cc:769] Sending connection context
0925 22:54:58.806651 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 22:54:58.806929 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.195@KRBTEST.COM" created.
I20260925 22:54:58.809271 29486 heartbeater.cc:347] Connected to a master server at 127.20.114.254:43439
I20260925 22:54:58.809535 29486 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:58.810046 29486 heartbeater.cc:511] Master 127.20.114.254:43439 requested a full tablet report, sending...
I20260925 22:54:58.811257 29161 ts_manager.cc:194] Registered new tserver with Master: 807ca216fd9745aa9a5f2d014e667762 (127.20.114.194:39777)
I20260925 22:54:58.811903 29161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:43273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
I20260925 22:54:58.825294 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:43439
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:42447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:58.937160 29491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:58.937443 29491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:58.937520 29491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:58.941516 29491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:58.941587 29491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:58.941668 29491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:54:58.946903 29491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:42447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:43439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:58.947938 29491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:58.949021 29491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:58.958664 29496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:58.958706 29497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:58.958741 29491 server_base.cc:1034] running on GCE node
W20260925 22:54:58.958886 29499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:58.959251 29491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:58.959892 29491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:58.961045 29491 hybrid_clock.cc:648] HybridClock initialized: now 1790376898961030 us; error 31 us; skew 500 ppm
Sep 25 22:54:58 dist-test-slave-x1lh krb5kdc[29116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376898, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:58.963656 29491 init.cc:377] Logged in from keytab as kudu/127.20.114.195@KRBTEST.COM (short username kudu)
I20260925 22:54:58.964531 29491 webserver.cc:460] Webserver started at http://127.20.114.195:44523/ using document root <none> and password file <none>
I20260925 22:54:58.965174 29491 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:58.965276 29491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:58.965535 29491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:58.967545 29491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "01029eacdecc45dfa7ed11d5ac69af0c"
format_stamp: "Formatted at 2026-09-25 22:54:58 on dist-test-slave-x1lh"
server_key: "e314353b3364f3840db53fec67764dc5"
server_key_iv: "fb3bc61b704f458690b87fdbd3fc32f8"
server_key_version: "encryptionkey@0"
I20260925 22:54:58.968104 29491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "01029eacdecc45dfa7ed11d5ac69af0c"
format_stamp: "Formatted at 2026-09-25 22:54:58 on dist-test-slave-x1lh"
server_key: "e314353b3364f3840db53fec67764dc5"
server_key_iv: "fb3bc61b704f458690b87fdbd3fc32f8"
server_key_version: "encryptionkey@0"
I20260925 22:54:58.971840 29491 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:54:58.974319 29506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:58.975546 29491 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:54:58.975687 29491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "01029eacdecc45dfa7ed11d5ac69af0c"
format_stamp: "Formatted at 2026-09-25 22:54:58 on dist-test-slave-x1lh"
server_key: "e314353b3364f3840db53fec67764dc5"
server_key_iv: "fb3bc61b704f458690b87fdbd3fc32f8"
server_key_version: "encryptionkey@0"
I20260925 22:54:58.975808 29491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:58.990061 29491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:58.993137 29491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:58.993346 29491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:58.993938 29491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:58.994758 29491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:58.994828 29491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:58.994899 29491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:58.994935 29491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:59.003892 29491 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:43499
I20260925 22:54:59.003927 29619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:43499 every 8 connection(s)
I20260925 22:54:59.004786 29491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:54:59.011535 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 29491
I20260925 22:54:59.011670 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:54:59.012034 20939 external_mini_cluster.cc:1428] Setting key c93e1f11194ed9ae279f15c64d5c67ef
Sep 25 22:54:59 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376898, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:54:59.019138 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:59.006679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:56109 (local address 127.20.114.254:43439)
0925 22:54:59.006855 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:59.006860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:59.008024 (+ 1164us) server_negotiation.cc:407] Connection header received
0925 22:54:59.009040 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:59.009043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:59.009091 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:59.009168 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:59.010428 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.010949 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.012685 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.013055 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.013170 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:59.015917 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:59.015920 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:59.015922 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:59.015945 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:59.017316 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:59.017816 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:59.017819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:59.017820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:59.017890 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:59.018206 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:59.018208 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:59.018209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:59.018367 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:59.018446 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:59.018919 (+ 473us) server_negotiation.cc:299] Negotiation successful
0925 22:54:59.019016 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 22:54:59.020021 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:59.007049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.195:56109)
0925 22:54:59.007835 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:59.007898 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:59.008782 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:59.009346 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:59.009354 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:59.009795 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:59.010296 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:59.010312 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.011063 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.011066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:59.012574 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:59.012584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.013206 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.013210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:59.013475 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:59.014194 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:59.014215 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:59.015756 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:59.017420 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:59.017425 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:59.017437 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:59.017721 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:59.017972 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:59.017974 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:59.017976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:59.018127 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:59.018460 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:59.018465 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:59.018699 (+ 234us) client_negotiation.cc:769] Sending connection context
0925 22:54:59.018860 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 22:54:59.019134 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":116,"threads_started":1}
I20260925 22:54:59.021198 29620 heartbeater.cc:347] Connected to a master server at 127.20.114.254:43439
I20260925 22:54:59.021447 29620 heartbeater.cc:464] Registering TS with master...
I20260925 22:54:59.021927 29620 heartbeater.cc:511] Master 127.20.114.254:43439 requested a full tablet report, sending...
I20260925 22:54:59.022987 29161 ts_manager.cc:194] Registered new tserver with Master: 01029eacdecc45dfa7ed11d5ac69af0c (127.20.114.195:43499)
I20260925 22:54:59.023567 29161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:56109
I20260925 22:54:59.026084 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.114.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.114.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.114.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.193.keytab.
Entry for principal oryx/127.20.114.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.193.keytab.
WARNING: no policy specified for oryx/127.20.114.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.114.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.114.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.194.keytab.
Entry for principal oryx/127.20.114.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.194.keytab.
WARNING: no policy specified for oryx/127.20.114.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.114.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.114.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.195.keytab.
Entry for principal oryx/127.20.114.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.195.keytab.
I20260925 22:54:59.099128 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 29223
I20260925 22:54:59.104723 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 29357
I20260925 22:54:59.110455 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 29491
I20260925 22:54:59.115782 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 29129
I20260925 22:54:59.122012 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:43439
--webserver_interface=127.20.114.254
--webserver_port=40391
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:42447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:43439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:59.233925 29629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:59.234225 29629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:59.234297 29629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:59.238545 29629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:59.238631 29629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:59.238667 29629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:59.238724 29629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:59.238761 29629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:59.244104 29629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:42447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:43439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:43439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=40391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:59.245285 29629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:59.246328 29629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:59.254833 29637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:59.254984 29629 server_base.cc:1034] running on GCE node
W20260925 22:54:59.254824 29634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:59.254824 29635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:59.255614 29629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:59.256681 29629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:59.257843 29629 hybrid_clock.cc:648] HybridClock initialized: now 1790376899257845 us; error 46 us; skew 500 ppm
Sep 25 22:54:59 dist-test-slave-x1lh krb5kdc[29116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376899, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:59.260375 29629 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:54:59.261264 29629 webserver.cc:460] Webserver started at http://127.20.114.254:40391/ using document root <none> and password file <none>
I20260925 22:54:59.261781 29629 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:59.261852 29629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:59.265463 29629 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:59.267432 29644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:59.268615 29629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 22:54:59.268718 29629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "b08caa5462e14ada8510219d9c89243b"
format_stamp: "Formatted at 2026-09-25 22:54:58 on dist-test-slave-x1lh"
server_key: "9cc7ed331d1d3ae21aaf9b8dd8323234"
server_key_iv: "e5903e73570872e2b84c251f52f3db38"
server_key_version: "encryptionkey@0"
I20260925 22:54:59.269279 29629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:59.281514 29629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:59.284775 29629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:59.284991 29629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:59.291980 29629 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:43439
I20260925 22:54:59.292061 29696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:43439 every 8 connection(s)
I20260925 22:54:59.293136 29629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:54:59.297431 29697 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:54:59.297955 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 29629
I20260925 22:54:59.298375 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:34217
--local_ip_for_outbound_sockets=127.20.114.193
--tserver_master_addrs=127.20.114.254:43439
--webserver_port=34257
--webserver_interface=127.20.114.193
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:42447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.114.193
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:54:59.299242 29697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b: Bootstrap starting.
I20260925 22:54:59.316102 29697 log.cc:824] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:59.322817 29697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:54:59.323187 29697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b: Bootstrap complete.
I20260925 22:54:59.327356 29697 raft_consensus.cc:348] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08caa5462e14ada8510219d9c89243b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43439 } }
I20260925 22:54:59.327790 29697 raft_consensus.cc:729] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b08caa5462e14ada8510219d9c89243b, State: Initialized, Role: FOLLOWER
I20260925 22:54:59.328269 29697 consensus_queue.cc:260] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08caa5462e14ada8510219d9c89243b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43439 } }
I20260925 22:54:59.328413 29697 raft_consensus.cc:388] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:54:59.328497 29697 raft_consensus.cc:482] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:54:59.328616 29697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:54:59.330595 29697 raft_consensus.cc:504] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08caa5462e14ada8510219d9c89243b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43439 } }
I20260925 22:54:59.330935 29697 leader_election.cc:302] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b08caa5462e14ada8510219d9c89243b; no voters:
I20260925 22:54:59.331281 29697 leader_election.cc:288] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:54:59.331370 29701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:54:59.331605 29701 raft_consensus.cc:686] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [term 2 LEADER]: Becoming Leader. State: Replica: b08caa5462e14ada8510219d9c89243b, State: Running, Role: LEADER
I20260925 22:54:59.331950 29701 consensus_queue.cc:237] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08caa5462e14ada8510219d9c89243b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43439 } }
I20260925 22:54:59.332512 29697 sys_catalog.cc:564] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:59.333501 29702 sys_catalog.cc:455] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b08caa5462e14ada8510219d9c89243b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08caa5462e14ada8510219d9c89243b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43439 } } }
I20260925 22:54:59.333506 29703 sys_catalog.cc:455] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b08caa5462e14ada8510219d9c89243b. Latest consensus state: current_term: 2 leader_uuid: "b08caa5462e14ada8510219d9c89243b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08caa5462e14ada8510219d9c89243b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43439 } } }
I20260925 22:54:59.333642 29702 sys_catalog.cc:458] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:59.333644 29703 sys_catalog.cc:458] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:59.334215 29710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:54:59.337596 29710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:54:59.338263 29710 catalog_manager.cc:1259] Loaded cluster ID: d71332ca992e423c95b92828f9ec9189
I20260925 22:54:59.338326 29710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:59.340664 29710 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:59.341358 29710 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b08caa5462e14ada8510219d9c89243b: Loaded TSK: 0
I20260925 22:54:59.342556 29710 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:54:59.414667 29699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:59.414924 29699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:59.414968 29699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:59.419013 29699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:59.419066 29699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:59.419158 29699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:54:59.424315 29699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:42447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.193.keytab
--principal=oryx/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:34217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=34257
--tserver_master_addrs=127.20.114.254:43439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:59.425356 29699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:59.426329 29699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:59.436700 29724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:59.436712 29723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:59.436702 29726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:59.437254 29699 server_base.cc:1034] running on GCE node
I20260925 22:54:59.437626 29699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:59.438269 29699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:59.439458 29699 hybrid_clock.cc:648] HybridClock initialized: now 1790376899439428 us; error 46 us; skew 500 ppm
Sep 25 22:54:59 dist-test-slave-x1lh krb5kdc[29116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376899, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:59.442384 29699 init.cc:377] Logged in from keytab as oryx/127.20.114.193@KRBTEST.COM (short username oryx)
I20260925 22:54:59.443225 29699 webserver.cc:460] Webserver started at http://127.20.114.193:34257/ using document root <none> and password file <none>
I20260925 22:54:59.443795 29699 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:59.443905 29699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:59.447889 29699 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:54:59.450119 29733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:59.451520 29699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:54:59.451673 29699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "e20d03800a2240cab16241b1fe73f18c"
format_stamp: "Formatted at 2026-09-25 22:54:58 on dist-test-slave-x1lh"
server_key: "5201aaa9b49d21b78f754f1757c8cc6f"
server_key_iv: "f60d1ef9721db8d4fb60be6ebad44895"
server_key_version: "encryptionkey@0"
I20260925 22:54:59.452205 29699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:59.472105 29699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:59.475857 29699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:59.476063 29699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:59.476667 29699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:59.477622 29699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:59.477694 29699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:59.477774 29699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:59.477820 29699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:59.486649 29699 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:34217
I20260925 22:54:59.486728 29846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:34217 every 8 connection(s)
I20260925 22:54:59.487694 29699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:54:59.495149 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 29699
I20260925 22:54:59.495558 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:39777
--local_ip_for_outbound_sockets=127.20.114.194
--tserver_master_addrs=127.20.114.254:43439
--webserver_port=42625
--webserver_interface=127.20.114.194
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:42447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.114.194
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:54:59 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:54:59 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:54:59.501127 29850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:59.489964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.193:50829)
0925 22:54:59.490659 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:59.490712 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:59.491549 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:59.492610 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:59.492617 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:59.493053 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:59.493514 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:59.493528 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.494754 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.494756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:59.495762 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:59.495776 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.496859 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.496864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:59.497111 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:59.497758 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:59.497786 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:59.500298 (+ 2512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":144,"threads_started":1}
W20260925 22:54:59.501631 29849 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:59.489849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:50829 (local address 127.20.114.254:43439)
0925 22:54:59.490284 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:59.490299 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:59.490839 (+ 540us) server_negotiation.cc:407] Connection header received
0925 22:54:59.491932 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:59.491943 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:59.492254 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:59.492525 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:59.493680 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.494633 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.495883 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.496683 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.496892 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:59.500611 (+ 3719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.193:50829: BlockingRecv error: recv got EOF from 127.20.114.193:50829 (error 108)
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":122,"threads_started":1}
W20260925 22:54:59.501767 29847 heartbeater.cc:644] Failed to heartbeat to 127.20.114.254:43439 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.114.254:43439: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Sep 25 22:54:59 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:54:59 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:54:59.507674 29850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:59.502283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.193:33479)
0925 22:54:59.502471 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:59.502491 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:59.502553 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:59.502809 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:59.502811 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:59.502933 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:59.503021 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:59.503026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.504126 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.504127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:59.504654 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:59.504661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.505328 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.505330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:59.505397 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:59.505889 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:59.505901 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:59.507582 (+ 1681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122}
W20260925 22:54:59.507855 29849 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:59.502404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:33479 (local address 127.20.114.254:43439)
0925 22:54:59.502549 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:59.502556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:59.502574 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:54:59.502628 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:59.502634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:59.502694 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:59.502792 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:59.503132 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.504020 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.504765 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.505218 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.505341 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:59.507743 (+ 2402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.193:33479: BlockingRecv error: recv got EOF from 127.20.114.193:33479 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 22:54:59 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:54:59 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:54:59.513509 29850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:59.508325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.193:35977)
0925 22:54:59.508464 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:59.508478 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:59.508548 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:59.508770 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:59.508773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:59.508951 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:59.509032 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:59.509038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.509700 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.509703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:59.510568 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:59.510578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.510986 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.510989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:59.511070 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:59.511494 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:59.511508 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:59.513386 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 22:54:59.513594 29849 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:59.508420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:35977 (local address 127.20.114.254:43439)
0925 22:54:59.508525 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:59.508528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:59.508539 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:54:59.508633 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:59.508636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:59.508671 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:59.508742 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:59.509131 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.509604 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.510683 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.510888 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.510961 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:59.513484 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.193:35977: BlockingRecv error: recv got EOF from 127.20.114.193:35977 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 22:54:59.513935 29847 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:54:59.609072 29851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:59.609338 29851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:59.609412 29851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:59.613399 29851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:59.613457 29851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:59.613561 29851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:54:59.618808 29851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:42447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.194.keytab
--principal=oryx/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:39777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=42625
--tserver_master_addrs=127.20.114.254:43439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:59.620090 29851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:59.621197 29851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:59.630561 29857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:59.630604 29859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:59.630708 29856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:59.630879 29851 server_base.cc:1034] running on GCE node
I20260925 22:54:59.631278 29851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:59.632081 29851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:59.633229 29851 hybrid_clock.cc:648] HybridClock initialized: now 1790376899633193 us; error 44 us; skew 500 ppm
Sep 25 22:54:59 dist-test-slave-x1lh krb5kdc[29116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376899, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:59.636112 29851 init.cc:377] Logged in from keytab as oryx/127.20.114.194@KRBTEST.COM (short username oryx)
I20260925 22:54:59.637130 29851 webserver.cc:460] Webserver started at http://127.20.114.194:42625/ using document root <none> and password file <none>
I20260925 22:54:59.637688 29851 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:59.637791 29851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:59.641610 29851 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 22:54:59.643599 29866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:59.644837 29851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:59.644985 29851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "807ca216fd9745aa9a5f2d014e667762"
format_stamp: "Formatted at 2026-09-25 22:54:58 on dist-test-slave-x1lh"
server_key: "b6220768f98836710c98c7e9bebbc2e8"
server_key_iv: "62314798f843857d290e7b6f3783752d"
server_key_version: "encryptionkey@0"
I20260925 22:54:59.645538 29851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:59.670015 29851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:59.674042 29851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:59.674240 29851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:59.674834 29851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:59.675671 29851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:59.675724 29851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:59.675791 29851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:59.675823 29851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:59.684619 29851 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:39777
I20260925 22:54:59.684679 29979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:39777 every 8 connection(s)
I20260925 22:54:59.685849 29851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:54:59.692648 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 29851
I20260925 22:54:59.693140 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:43499
--local_ip_for_outbound_sockets=127.20.114.195
--tserver_master_addrs=127.20.114.254:43439
--webserver_port=44523
--webserver_interface=127.20.114.195
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:42447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.114.195
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:54:59 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:54:59 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:54:59.697000 29849 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:59.687806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:55015 (local address 127.20.114.254:43439)
0925 22:54:59.687943 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:59.687947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:59.689061 (+ 1114us) server_negotiation.cc:407] Connection header received
0925 22:54:59.689926 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:59.689929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:59.689977 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:59.690049 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:59.691179 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.691707 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.692800 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.693115 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.693231 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:59.696806 (+ 3575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.194:55015: BlockingRecv error: recv got EOF from 127.20.114.194:55015 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:54:59.697546 29982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:59.688222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.194:55015)
0925 22:54:59.688857 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:59.688973 (+ 116us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:59.689749 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:59.690211 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:59.690217 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:59.690570 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:59.691057 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:59.691071 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.691811 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.691814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:59.692660 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:59.692668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.693269 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.693273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:59.693472 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:59.694106 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:59.694122 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:59.696612 (+ 2490us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":106,"threads_started":1}
W20260925 22:54:59.698109 29980 heartbeater.cc:644] Failed to heartbeat to 127.20.114.254:43439 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.114.254:43439: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Sep 25 22:54:59 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:54:59 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:54:59.703863 29982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:59.698584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.194:52717)
0925 22:54:59.698693 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:59.698707 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:59.698775 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:59.699063 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:59.699065 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:59.699196 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:59.699275 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:59.699279 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.700352 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.700354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:59.700908 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:59.700917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.701576 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.701578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:59.701639 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:59.702144 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:59.702164 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:59.703779 (+ 1615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 22:54:59.704116 29849 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:59.698681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:52717 (local address 127.20.114.254:43439)
0925 22:54:59.698822 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:59.698829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:59.698847 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:54:59.698895 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:59.698901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:59.698956 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:59.699055 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:59.699382 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.700245 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.701023 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.701460 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.701587 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:59.703966 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.194:52717: BlockingRecv error: recv got EOF from 127.20.114.194:52717 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 22:54:59 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:54:59 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:54:59.709234 29982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:59.704470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.194:52471)
0925 22:54:59.704552 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:59.704567 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:59.704630 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:59.704831 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:59.704833 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:59.704993 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:59.705056 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:59.705060 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.706043 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.706045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:59.706535 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:59.706542 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.707072 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.707075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:59.707127 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:59.707587 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:59.707597 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:59.709146 (+ 1549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 22:54:59.709400 29849 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:59.704545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:52471 (local address 127.20.114.254:43439)
0925 22:54:59.704642 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:59.704646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:59.704659 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:54:59.704694 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:59.704696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:59.704735 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:59.704813 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:59.705154 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.705945 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.706633 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.706968 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.707062 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:59.709292 (+ 2230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.194:52471: BlockingRecv error: recv got EOF from 127.20.114.194:52471 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 22:54:59.709610 29980 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:54:59.805420 29983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:59.805696 29983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:59.805739 29983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:59.809772 29983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:59.809828 29983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:59.810014 29983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:54:59.815522 29983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:42447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/krb5kdc/oryx_127.20.114.195.keytab
--principal=oryx/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:43499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=44523
--tserver_master_addrs=127.20.114.254:43439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:59.816576 29983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:59.817734 29983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:59.827008 29991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:59.827169 29983 server_base.cc:1034] running on GCE node
W20260925 22:54:59.827041 29989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:59.827008 29988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:59.827760 29983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:59.828362 29983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:59.829545 29983 hybrid_clock.cc:648] HybridClock initialized: now 1790376899829525 us; error 36 us; skew 500 ppm
Sep 25 22:54:59 dist-test-slave-x1lh krb5kdc[29116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376899, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:59.832386 29983 init.cc:377] Logged in from keytab as oryx/127.20.114.195@KRBTEST.COM (short username oryx)
I20260925 22:54:59.833436 29983 webserver.cc:460] Webserver started at http://127.20.114.195:44523/ using document root <none> and password file <none>
I20260925 22:54:59.834132 29983 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:59.834239 29983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:59.838250 29983 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:54:59.840279 29998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:59.841603 29983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:59.841699 29983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "01029eacdecc45dfa7ed11d5ac69af0c"
format_stamp: "Formatted at 2026-09-25 22:54:58 on dist-test-slave-x1lh"
server_key: "e314353b3364f3840db53fec67764dc5"
server_key_iv: "fb3bc61b704f458690b87fdbd3fc32f8"
server_key_version: "encryptionkey@0"
I20260925 22:54:59.842213 29983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:59.857266 29983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:59.860749 29983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:59.860985 29983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:59.861578 29983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:54:59.862365 29983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:54:59.862411 29983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:59.862452 29983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:54:59.862473 29983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:59.871492 29983 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:43499
I20260925 22:54:59.871515 30111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:43499 every 8 connection(s)
I20260925 22:54:59.872588 29983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:54:59.879724 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 29983
Sep 25 22:54:59 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:54:59 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:54:59.884395 29849 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:59.874558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:42779 (local address 127.20.114.254:43439)
0925 22:54:59.874729 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:59.874734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:59.876051 (+ 1317us) server_negotiation.cc:407] Connection header received
0925 22:54:59.876925 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:59.876934 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:59.876981 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:59.877055 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:59.878281 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.878793 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.880170 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.880500 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.880615 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:59.884270 (+ 3655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.195:42779: BlockingRecv error: recv got EOF from 127.20.114.195:42779 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 22:54:59.885178 30114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:59.875026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.195:42779)
0925 22:54:59.875896 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:59.875937 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:59.876694 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:59.877219 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:59.877226 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:59.877567 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:59.878045 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:59.878058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.878906 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.878909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:59.880033 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:59.880043 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.880674 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.880681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:59.880903 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:59.881558 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:59.881581 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:59.884165 (+ 2584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":542,"threads_started":1}
W20260925 22:54:59.885689 30112 heartbeater.cc:644] Failed to heartbeat to 127.20.114.254:43439 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.114.254:43439: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
I20260925 22:54:59.889860 30116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:59.880736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39706 (local address 127.20.114.254:43439)
0925 22:54:59.880991 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:59.880996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:59.881009 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:54:59.881087 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:59.881091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:59.881176 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:59.881276 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:59.882219 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.882795 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.883746 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.883987 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.884094 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:59.885662 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:59.885669 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:59.885688 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:59.885724 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:59.887602 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:59.888074 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:59.888080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:59.888084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:59.888201 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:59.888498 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:59.888502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:59.888504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:59.888952 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:59.889163 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:59.889421 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 22:54:59.889669 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":184,"thread_start_us":67,"threads_started":1}
Sep 25 22:54:59 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:54:59 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:54:59.891232 30114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:59.886215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.195:41845)
0925 22:54:59.886404 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:59.886420 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:59.886493 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:59.886722 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:59.886725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:59.886889 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:59.886981 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:59.886986 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.887680 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.887684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:59.888221 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:59.888228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.888648 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.888651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:59.888749 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:59.889222 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:59.889236 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:59.891120 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102}
W20260925 22:54:59.891352 29849 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:59.886306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:41845 (local address 127.20.114.254:43439)
0925 22:54:59.886463 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:59.886467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:59.886480 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:54:59.886568 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:59.886572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:59.886615 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:59.886712 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:59.887087 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.887571 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.888332 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.888545 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.888692 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:59.891227 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.195:41845: BlockingRecv error: recv got EOF from 127.20.114.195:41845 (error 108)
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":57}
Sep 25 22:54:59 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:54:59 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:54:59.896421 30114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:59.891857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.195:45765)
0925 22:54:59.891972 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:59.891987 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:59.892058 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:59.892272 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:59.892275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:59.892429 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:59.892492 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:59.892497 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.893289 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.893292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:59.893813 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:59.893821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.894253 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.894256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:59.894334 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:59.894728 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:59.894747 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:59.896287 (+ 1540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 22:54:59.896484 29849 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:59.891952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:45765 (local address 127.20.114.254:43439)
0925 22:54:59.892068 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:59.892072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:59.892085 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:54:59.892126 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:59.892130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:59.892164 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:59.892262 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:59.892599 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.893174 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.893934 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:59.894146 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:59.894260 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:59.896385 (+ 2125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.195:45765: BlockingRecv error: recv got EOF from 127.20.114.195:45765 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 22:54:59.896647 30112 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 22:55:00 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:00 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:00.520117 30117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:00.514622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.193:53751)
0925 22:55:00.514906 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:00.514923 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:00.515017 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:00.515286 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:00.515289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:00.515478 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:00.515604 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:00.515610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:00.516363 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:00.516365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:00.516980 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:00.516990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:00.517416 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:00.517418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:00.517488 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:00.518021 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:00.518033 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:00.520000 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":85,"threads_started":1}
W20260925 22:55:00.520313 30118 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:00.514713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:53751 (local address 127.20.114.254:43439)
0925 22:55:00.514967 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:00.514971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:00.514984 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:55:00.515110 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:00.515112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:00.515155 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:00.515238 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:00.515710 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:00.516257 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:00.517086 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:00.517316 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:00.517395 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:00.520163 (+ 2768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.193:53751: BlockingRecv error: recv got EOF from 127.20.114.193:53751 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":78,"threads_started":1}
Sep 25 22:55:00 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:00 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:00.716022 30119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:00.710424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.194:43483)
0925 22:55:00.710706 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:00.710720 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:00.710826 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:00.711052 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:00.711054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:00.711233 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:00.711325 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:00.711329 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:00.712223 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:00.712229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:00.712802 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:00.712811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:00.713344 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:00.713348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:00.713429 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:00.714004 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:00.714017 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:00.715902 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":109,"threads_started":1}
W20260925 22:55:00.716281 30118 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:00.710493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:43483 (local address 127.20.114.254:43439)
0925 22:55:00.710647 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:00.710653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:00.710779 (+ 126us) server_negotiation.cc:407] Connection header received
0925 22:55:00.710898 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:00.710900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:00.710945 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:00.711014 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:00.711552 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:00.712102 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:00.712997 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:00.713228 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:00.713345 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:00.716044 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.194:43483: BlockingRecv error: recv got EOF from 127.20.114.194:43483 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 22:55:00 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:00 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:00.902637 30120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:00.897311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.195:42283)
0925 22:55:00.897590 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:00.897603 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:00.897734 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:00.897959 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:00.897961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:00.898137 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:00.898231 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:00.898237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:00.898955 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:00.898958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:00.899539 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:00.899547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:00.899957 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:00.899959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:00.900017 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:00.900566 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:00.900577 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:00.902505 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":109,"threads_started":1}
W20260925 22:55:00.902770 30118 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:00.897398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:42283 (local address 127.20.114.254:43439)
0925 22:55:00.897544 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:00.897549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:00.897682 (+ 133us) server_negotiation.cc:407] Connection header received
0925 22:55:00.897802 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:00.897804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:00.897849 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:00.897918 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:00.898330 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:00.898850 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:00.899639 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:00.899860 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:00.899937 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:00.902638 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.195:42283: BlockingRecv error: recv got EOF from 127.20.114.195:42283 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 22:55:01 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:01 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:01.526865 30121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:01.520994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.193:46667)
0925 22:55:01.521275 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:01.521289 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:01.521382 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:01.521693 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:01.521695 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:01.521870 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:01.521964 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:01.521968 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:01.522805 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:01.522809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:01.523462 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:01.523474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:01.524112 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:01.524117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:01.524210 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:01.524834 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:01.524847 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:01.526748 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":98,"threads_started":1}
W20260925 22:55:01.527094 30122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:01.521160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:46667 (local address 127.20.114.254:43439)
0925 22:55:01.521417 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:01.521421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:01.521435 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:55:01.521477 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:01.521479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:01.521519 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:01.521648 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:01.522107 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:01.522656 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:01.523667 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:01.523978 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:01.524080 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:01.526925 (+ 2845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.193:46667: BlockingRecv error: recv got EOF from 127.20.114.193:46667 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":116,"threads_started":1}
Sep 25 22:55:01 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:01 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:01.722426 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:01.716935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.194:49973)
0925 22:55:01.717218 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:01.717232 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:01.717331 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:01.717540 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:01.717542 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:01.717713 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:01.717802 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:01.717807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:01.718529 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:01.718532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:01.719078 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:01.719086 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:01.719493 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:01.719496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:01.719552 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:01.720120 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:01.720138 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:01.722288 (+ 2150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":105,"threads_started":1}
W20260925 22:55:01.722510 30122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:01.716975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:49973 (local address 127.20.114.254:43439)
0925 22:55:01.717123 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:01.717128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:01.717288 (+ 160us) server_negotiation.cc:407] Connection header received
0925 22:55:01.717398 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:01.717400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:01.717441 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:01.717506 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:01.717900 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:01.718431 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:01.719186 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:01.719402 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:01.719475 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:01.722421 (+ 2946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.194:49973: BlockingRecv error: recv got EOF from 127.20.114.194:49973 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 22:55:01 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:01 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:01.909170 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:01.903472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.195:46487)
0925 22:55:01.903795 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:01.903810 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:01.903911 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:01.904157 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:01.904160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:01.904333 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:01.904424 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:01.904429 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:01.905189 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:01.905192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:01.905970 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:01.905980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:01.906507 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:01.906511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:01.906601 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:01.907075 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:01.907087 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:01.909024 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":108,"threads_started":1}
W20260925 22:55:01.909282 30122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:01.903664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:46487 (local address 127.20.114.254:43439)
0925 22:55:01.903817 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:01.903820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:01.903862 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:55:01.903993 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:01.903995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:01.904039 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:01.904118 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:01.904517 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:01.905061 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:01.906089 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:01.906395 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:01.906504 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:01.909152 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.195:46487: BlockingRecv error: recv got EOF from 127.20.114.195:46487 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 22:55:02 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:02 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:02.547688 30125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:02.527624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.193:52131)
0925 22:55:02.527916 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:02.527929 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:02.528026 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:02.528278 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:02.528280 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:02.528456 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:02.528553 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:02.528558 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:02.529300 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:02.529302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:02.529876 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:02.529884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:02.530296 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:02.530298 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:02.530360 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:02.545366 (+ 15006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:02.545386 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:02.547499 (+ 2113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":117,"threads_started":1}
W20260925 22:55:02.547765 30126 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:02.527682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:52131 (local address 127.20.114.254:43439)
0925 22:55:02.527930 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:02.527934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:02.527998 (+ 64us) server_negotiation.cc:407] Connection header received
0925 22:55:02.528111 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:02.528113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:02.528154 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:02.528233 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:02.528651 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:02.529209 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:02.529974 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:02.530193 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:02.530265 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:02.547626 (+ 17361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.193:52131: BlockingRecv error: recv got EOF from 127.20.114.193:52131 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":92,"threads_started":1}
Sep 25 22:55:02 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:02 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:02.728353 30127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:02.723229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.194:49113)
0925 22:55:02.723543 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:02.723557 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:02.723654 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:02.723903 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:02.723905 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:02.724078 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:02.724171 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:02.724176 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:02.724895 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:02.724897 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:02.725440 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:02.725447 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:02.725849 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:02.725851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:02.725911 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:02.726397 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:02.726409 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:02.728239 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":114,"threads_started":1}
W20260925 22:55:02.728541 30126 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:02.723307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:49113 (local address 127.20.114.254:43439)
0925 22:55:02.723465 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:02.723470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:02.723627 (+ 157us) server_negotiation.cc:407] Connection header received
0925 22:55:02.723757 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:02.723759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:02.723803 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:02.723868 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:02.724271 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:02.724779 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:02.725537 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:02.725757 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:02.725832 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:02.728380 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.194:49113: BlockingRecv error: recv got EOF from 127.20.114.194:49113 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 22:55:02 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:02 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:02.915344 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:02.910121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.195:36961)
0925 22:55:02.910425 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:02.910438 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:02.910534 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:02.910815 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:02.910817 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:02.910991 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:02.911080 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:02.911085 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:02.911787 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:02.911790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:02.912333 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:02.912341 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:02.912741 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:02.912744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:02.912801 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:02.913322 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:02.913333 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:02.915196 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":110,"threads_started":1}
W20260925 22:55:02.915553 30126 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:02.910215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:36961 (local address 127.20.114.254:43439)
0925 22:55:02.910365 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:02.910372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:02.910508 (+ 136us) server_negotiation.cc:407] Connection header received
0925 22:55:02.910653 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:02.910656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:02.910706 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:02.910778 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:02.911179 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:02.911691 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:02.912435 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:02.912647 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:02.912724 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:02.915381 (+ 2657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.195:36961: BlockingRecv error: recv got EOF from 127.20.114.195:36961 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 22:55:03 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:03 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:03.553686 30129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:03.548661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.193:49509)
0925 22:55:03.548942 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:03.548955 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:03.549046 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:03.549274 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:03.549276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:03.549446 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:03.549567 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:03.549572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:03.550295 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:03.550297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:03.550841 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:03.550848 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:03.551240 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:03.551242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:03.551298 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:03.551759 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:03.551770 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:03.553561 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":86,"threads_started":1}
W20260925 22:55:03.553824 30130 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:03.548691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:49509 (local address 127.20.114.254:43439)
0925 22:55:03.548948 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:03.548952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:03.549003 (+ 51us) server_negotiation.cc:407] Connection header received
0925 22:55:03.549125 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:03.549127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:03.549170 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:03.549244 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:03.549674 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:03.550203 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:03.550936 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:03.551151 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:03.551222 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:03.553690 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.193:49509: BlockingRecv error: recv got EOF from 127.20.114.193:49509 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":87,"threads_started":1}
Sep 25 22:55:03 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:03 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:03.734835 30131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:03.729190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.194:55711)
0925 22:55:03.729499 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:03.729528 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:03.729665 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:03.729976 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:03.729979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:03.730228 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:03.730379 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:03.730389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:03.731135 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:03.731137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:03.731699 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:03.731706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:03.732097 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:03.732100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:03.732160 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:03.732707 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:03.732726 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:03.734703 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":84,"threads_started":1}
W20260925 22:55:03.735037 30130 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:03.729282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:55711 (local address 127.20.114.254:43439)
0925 22:55:03.729465 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:03.729469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:03.729614 (+ 145us) server_negotiation.cc:407] Connection header received
0925 22:55:03.729786 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:03.729790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:03.729843 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:03.729916 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:03.730499 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:03.731040 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:03.731797 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:03.732007 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:03.732079 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:03.734888 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.194:55711: BlockingRecv error: recv got EOF from 127.20.114.194:55711 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 22:55:03 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:03 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:03.921330 30132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:03.916166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.195:59191)
0925 22:55:03.916428 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:03.916442 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:03.916541 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:03.916774 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:03.916776 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:03.916970 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:03.917064 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:03.917069 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:03.917780 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:03.917782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:03.918325 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:03.918332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:03.918725 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:03.918728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:03.918785 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:03.919350 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:03.919362 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:03.921209 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":67,"threads_started":1}
W20260925 22:55:03.921452 30130 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:03.916252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:59191 (local address 127.20.114.254:43439)
0925 22:55:03.916425 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:03.916429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:03.916493 (+ 64us) server_negotiation.cc:407] Connection header received
0925 22:55:03.916606 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:03.916608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:03.916650 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:03.916727 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:03.917174 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:03.917690 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:03.918430 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:03.918633 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:03.918715 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:03.921314 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.195:59191: BlockingRecv error: recv got EOF from 127.20.114.195:59191 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 25 22:55:04 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:04 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:04.559768 30133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:04.554383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.193:60511)
0925 22:55:04.554656 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:04.554679 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:04.554762 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:04.555002 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:04.555004 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:04.555180 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:04.555351 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:04.555358 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:04.556117 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:04.556119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:04.556745 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:04.556754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:04.557187 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:04.557189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:04.557250 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:04.557818 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:04.557830 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:04.559652 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":69,"threads_started":1}
W20260925 22:55:04.559901 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:04.554466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:60511 (local address 127.20.114.254:43439)
0925 22:55:04.554748 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:04.554751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:04.554764 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:55:04.554852 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:04.554854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:04.554894 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:04.554968 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:04.555450 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:04.556026 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:04.556848 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:04.557087 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:04.557163 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:04.559767 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.193:60511: BlockingRecv error: recv got EOF from 127.20.114.193:60511 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":103,"threads_started":1}
Sep 25 22:55:04 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:04 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:04.741153 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:04.735600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.194:33809)
0925 22:55:04.735881 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:04.735894 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:04.735986 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:04.736215 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:04.736217 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:04.736400 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:04.736489 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:04.736494 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:04.737283 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:04.737286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:04.737922 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:04.737932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:04.738446 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:04.738449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:04.738522 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:04.739106 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:04.739119 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:04.740987 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":112,"threads_started":1}
W20260925 22:55:04.741214 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:04.735726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:33809 (local address 127.20.114.254:43439)
0925 22:55:04.735856 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:04.735860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:04.735959 (+ 99us) server_negotiation.cc:407] Connection header received
0925 22:55:04.736064 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:04.736066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:04.736107 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:04.736176 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:04.736579 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:04.737193 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:04.738041 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:04.738333 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:04.738446 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:04.741107 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.194:33809: BlockingRecv error: recv got EOF from 127.20.114.194:33809 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 22:55:04 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:04 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:04.928198 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:04.922214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.195:60207)
0925 22:55:04.922514 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:04.922528 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:04.922661 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:04.922935 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:04.922938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:04.923149 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:04.923294 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:04.923302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:04.924578 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:04.924582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:04.925282 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:04.925292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:04.925712 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:04.925714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:04.925778 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:04.926320 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:04.926337 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:04.928074 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":103,"threads_started":1}
W20260925 22:55:04.928366 30134 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:04.922296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:60207 (local address 127.20.114.254:43439)
0925 22:55:04.922469 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:04.922473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:04.922606 (+ 133us) server_negotiation.cc:407] Connection header received
0925 22:55:04.922772 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:04.922776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:04.922830 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:04.922917 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:04.923461 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:04.924463 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:04.925394 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:04.925618 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:04.925698 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:04.928233 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.195:60207: BlockingRecv error: recv got EOF from 127.20.114.195:60207 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 25 22:55:05 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:05 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:05.566234 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:05.560584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.193:36921)
0925 22:55:05.560964 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:05.560978 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:05.561077 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:05.561316 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:05.561318 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:05.561492 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:05.561642 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:05.561649 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:05.562460 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:05.562465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:05.563243 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:05.563253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:05.563667 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:05.563669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:05.563726 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:05.564261 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:05.564274 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:05.566118 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":103,"threads_started":1}
W20260925 22:55:05.566377 30138 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:05.560711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:36921 (local address 127.20.114.254:43439)
0925 22:55:05.560976 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:05.560979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:05.561053 (+ 74us) server_negotiation.cc:407] Connection header received
0925 22:55:05.561160 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:05.561163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:05.561207 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:05.561283 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:05.561794 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:05.562339 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:05.563349 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:05.563575 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:05.563652 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:05.566242 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.193:36921: BlockingRecv error: recv got EOF from 127.20.114.193:36921 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":57,"threads_started":1}
Sep 25 22:55:05 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:05 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:05.747543 30139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:05.742003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.194:52841)
0925 22:55:05.742297 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:05.742310 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:05.742412 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:05.742663 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:05.742666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:05.742876 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:05.742966 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:05.742972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:05.743690 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:05.743692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:05.744224 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:05.744231 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:05.744689 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:05.744691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:05.744750 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:05.745306 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:05.745317 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:05.747389 (+ 2072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":87,"threads_started":1}
W20260925 22:55:05.747627 30138 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:05.742146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:52841 (local address 127.20.114.254:43439)
0925 22:55:05.742313 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:05.742317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:05.742379 (+ 62us) server_negotiation.cc:407] Connection header received
0925 22:55:05.742489 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:05.742491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:05.742533 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:05.742600 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:05.743072 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:05.743602 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:05.744346 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:05.744585 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:05.744674 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:05.747489 (+ 2815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.194:52841: BlockingRecv error: recv got EOF from 127.20.114.194:52841 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 22:55:05 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:05 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:05.934945 30140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:05.929053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.195:34975)
0925 22:55:05.929390 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:05.929406 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:05.929562 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:05.929833 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:05.929836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:05.930050 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:05.930152 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:05.930158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:05.930911 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:05.930917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:05.931478 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:05.931485 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:05.931927 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:05.931930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:05.932034 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:05.932521 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:05.932533 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:05.934833 (+ 2300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":102,"threads_started":1}
W20260925 22:55:05.935082 30138 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:05.929202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:34975 (local address 127.20.114.254:43439)
0925 22:55:05.929367 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:05.929371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:05.929488 (+ 117us) server_negotiation.cc:407] Connection header received
0925 22:55:05.929684 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:05.929686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:05.929731 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:05.929814 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:05.930273 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:05.930801 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:05.931596 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:05.931816 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:05.931924 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:05.934956 (+ 3032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.195:34975: BlockingRecv error: recv got EOF from 127.20.114.195:34975 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 22:55:06 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:06 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:06.572717 30141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:06.567027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.193:40827)
0925 22:55:06.567337 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:06.567351 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:06.567443 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:06.567683 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:06.567686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:06.567865 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:06.567960 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:06.567966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:06.568697 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:06.568701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:06.569417 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:06.569430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:06.569888 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:06.569891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:06.569963 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:06.570471 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:06.570484 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:06.572472 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":64,"threads_started":1}
W20260925 22:55:06.572767 30142 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:06.567133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:40827 (local address 127.20.114.254:43439)
0925 22:55:06.567394 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:06.567397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:06.567410 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:55:06.567526 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:06.567528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:06.567569 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:06.567642 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:06.568055 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:06.568582 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:06.569545 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:06.569779 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:06.569857 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:06.572600 (+ 2743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.193:40827: BlockingRecv error: recv got EOF from 127.20.114.193:40827 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":76,"threads_started":1}
Sep 25 22:55:06 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:06 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:06.754341 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:06.748430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.194:43485)
0925 22:55:06.748781 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:06.748798 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:06.748951 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:06.749229 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:06.749232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:06.749440 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:06.749607 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:06.749615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:06.750442 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:06.750447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:06.751148 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:06.751157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:06.751650 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:06.751653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:06.751740 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:06.752241 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:06.752253 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:06.754172 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":74,"threads_started":1}
W20260925 22:55:06.754403 30142 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:06.748564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:43485 (local address 127.20.114.254:43439)
0925 22:55:06.748810 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:06.748814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:06.748871 (+ 57us) server_negotiation.cc:407] Connection header received
0925 22:55:06.749065 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:06.749069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:06.749124 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:06.749210 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:06.749707 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:06.750328 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:06.751268 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:06.751509 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:06.751645 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:06.754297 (+ 2652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.194:43485: BlockingRecv error: recv got EOF from 127.20.114.194:43485 (error 108)
Metrics: {"server-negotiator.queue_time_us":174}
Sep 25 22:55:06 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:06 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:06.941332 30144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:06.935935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.195:56235)
0925 22:55:06.936253 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:06.936266 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:06.936370 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:06.936628 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:06.936630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:06.936814 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:06.936932 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:06.936937 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:06.937733 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:06.937735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:06.938283 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:06.938290 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:06.938713 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:06.938715 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:06.938773 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:06.939351 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:06.939363 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:06.941207 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":104,"threads_started":1}
W20260925 22:55:06.941464 30142 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:06.935970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:56235 (local address 127.20.114.254:43439)
0925 22:55:06.936167 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:06.936171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:06.936348 (+ 177us) server_negotiation.cc:407] Connection header received
0925 22:55:06.936454 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:06.936456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:06.936503 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:06.936598 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:06.937042 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:06.937627 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:06.938398 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:06.938616 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:06.938695 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:06.941343 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.195:56235: BlockingRecv error: recv got EOF from 127.20.114.195:56235 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 25 22:55:07 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:07 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:07.584348 30145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:07.573641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.193:43659)
0925 22:55:07.573933 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:07.573949 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:07.574054 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:07.574348 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:07.574351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:07.574573 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:07.574705 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:07.574714 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:07.575576 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:07.575579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:07.576125 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:07.576133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:07.576636 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:07.576639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:07.576697 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:07.582272 (+ 5575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:07.582285 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:07.584229 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":118,"threads_started":1}
W20260925 22:55:07.584554 30146 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:07.573814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:43659 (local address 127.20.114.254:43439)
0925 22:55:07.574080 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:07.574084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:07.574099 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:55:07.574135 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:07.574138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:07.574201 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:07.574310 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:07.574829 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:07.575477 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:07.576262 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:07.576486 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:07.576620 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:07.584373 (+ 7753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.193:43659: BlockingRecv error: recv got EOF from 127.20.114.193:43659 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":71,"threads_started":1}
Sep 25 22:55:07 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:07 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:07.760998 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:07.755122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.194:37781)
0925 22:55:07.755400 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:07.755413 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:07.755538 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:07.755810 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:07.755813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:07.756037 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:07.756129 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:07.756134 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:07.757078 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:07.757083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:07.757715 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:07.757726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:07.758330 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:07.758332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:07.758420 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:07.759057 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:07.759068 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:07.760824 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":83,"threads_started":1}
W20260925 22:55:07.761130 30146 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:07.755181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:37781 (local address 127.20.114.254:43439)
0925 22:55:07.755342 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:07.755347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:07.755488 (+ 141us) server_negotiation.cc:407] Connection header received
0925 22:55:07.755652 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:07.755655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:07.755707 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:07.755790 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:07.756241 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:07.756978 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:07.757857 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:07.758208 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:07.758322 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:07.761010 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.194:37781: BlockingRecv error: recv got EOF from 127.20.114.194:37781 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 22:55:07 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:07 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:07.947613 30148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:07.942244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.195:59185)
0925 22:55:07.942602 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:07.942616 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:07.942758 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:07.943087 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:07.943091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:07.943355 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:07.943448 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:07.943453 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:07.944161 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:07.944163 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:07.944714 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:07.944720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:07.945132 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:07.945134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:07.945193 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:07.945650 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:07.945661 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:07.947499 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":74,"threads_started":1}
W20260925 22:55:07.947703 30146 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:07.942272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:59185 (local address 127.20.114.254:43439)
0925 22:55:07.942451 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:07.942455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:07.942694 (+ 239us) server_negotiation.cc:407] Connection header received
0925 22:55:07.942896 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:07.942899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:07.942958 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:07.943045 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:07.943554 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:07.944072 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:07.944817 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:07.945045 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:07.945152 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:07.947612 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.195:59185: BlockingRecv error: recv got EOF from 127.20.114.195:59185 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 25 22:55:08 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:08 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:08.590390 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:08.585157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.193:38413)
0925 22:55:08.585444 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:08.585458 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:08.585548 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:08.585774 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:08.585776 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:08.585949 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:08.586040 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:08.586045 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:08.586838 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:08.586841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:08.587397 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:08.587405 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:08.587815 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:08.587817 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:08.587876 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:08.588388 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:08.588400 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:08.590270 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":72,"threads_started":1}
W20260925 22:55:08.590595 30150 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:08.585207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:38413 (local address 127.20.114.254:43439)
0925 22:55:08.585540 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:08.585544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:08.585557 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:55:08.585615 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:08.585617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:08.585659 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:08.585735 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:08.586144 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:08.586730 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:08.587496 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:08.587719 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:08.587803 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:08.590419 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.193:38413: BlockingRecv error: recv got EOF from 127.20.114.193:38413 (error 108)
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":112,"threads_started":1}
Sep 25 22:55:08 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:08 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:08.767618 30151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:08.761859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.194:49829)
0925 22:55:08.762185 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:08.762200 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:08.762353 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:08.762622 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:08.762625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:08.762835 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:08.762932 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:08.762938 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:08.763685 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:08.763691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:08.764249 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:08.764256 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:08.764717 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:08.764720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:08.764799 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:08.765391 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:08.765402 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:08.767434 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":87,"threads_started":1}
W20260925 22:55:08.767690 30150 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:08.761981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:49829 (local address 127.20.114.254:43439)
0925 22:55:08.762155 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:08.762160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:08.762287 (+ 127us) server_negotiation.cc:407] Connection header received
0925 22:55:08.762455 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:08.762459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:08.762511 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:08.762597 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:08.763051 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:08.763574 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:08.764380 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:08.764605 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:08.764715 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:08.767562 (+ 2847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.194:49829: BlockingRecv error: recv got EOF from 127.20.114.194:49829 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 25 22:55:08 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:08 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:08.954305 30152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:08.948387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.195:33689)
0925 22:55:08.948677 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:08.948708 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:08.948854 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:08.949168 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:08.949171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:08.949397 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:08.949486 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:08.949492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:08.950350 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:08.950356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:08.950974 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:08.950983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:08.951443 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:08.951447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:08.951532 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:08.952041 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:08.952060 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:08.954121 (+ 2061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":89,"threads_started":1}
W20260925 22:55:08.954394 30150 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:08.948465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:33689 (local address 127.20.114.254:43439)
0925 22:55:08.948628 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:08.948633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:08.948779 (+ 146us) server_negotiation.cc:407] Connection header received
0925 22:55:08.949005 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:08.949008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:08.949056 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:08.949160 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:08.949619 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:08.950240 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:08.951103 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:08.951331 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:08.951442 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:08.954267 (+ 2825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.195:33689: BlockingRecv error: recv got EOF from 127.20.114.195:33689 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 22:55:09 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:09 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:09.596324 30153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:09.591232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.193:39381)
0925 22:55:09.591518 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:09.591530 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:09.591611 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:09.591835 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:09.591837 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:09.592010 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:09.592113 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:09.592118 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:09.592838 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:09.592841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:09.593461 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:09.593469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:09.593864 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:09.593867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:09.593926 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:09.594474 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:09.594486 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:09.596201 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":82,"threads_started":1}
W20260925 22:55:09.596477 30154 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:09.591290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:39381 (local address 127.20.114.254:43439)
0925 22:55:09.591569 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:09.591573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:09.591586 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:55:09.591680 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:09.591682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:09.591723 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:09.591796 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:09.592210 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:09.592748 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:09.593560 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:09.593772 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:09.593844 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:09.596342 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.193:39381: BlockingRecv error: recv got EOF from 127.20.114.193:39381 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":112,"threads_started":1}
Sep 25 22:55:09 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:09 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:09.774130 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:09.768393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.194:44659)
0925 22:55:09.768674 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:09.768696 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:09.768837 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:09.769128 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:09.769131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:09.769342 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:09.769431 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:09.769436 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:09.770198 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:09.770203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:09.770759 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:09.770767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:09.771229 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:09.771233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:09.771311 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:09.771881 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:09.771892 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:09.773956 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":83,"threads_started":1}
W20260925 22:55:09.774382 30154 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:09.768482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:44659 (local address 127.20.114.254:43439)
0925 22:55:09.768643 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:09.768648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:09.768772 (+ 124us) server_negotiation.cc:407] Connection header received
0925 22:55:09.768964 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:09.768967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:09.769022 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:09.769106 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:09.769548 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:09.770083 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:09.770886 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:09.771115 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:09.771226 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:09.774143 (+ 2917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.194:44659: BlockingRecv error: recv got EOF from 127.20.114.194:44659 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 22:55:09 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:09 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:09.961138 30156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:09.955225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.195:47033)
0925 22:55:09.955526 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:09.955542 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:09.955653 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:09.955983 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:09.955986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:09.956192 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:09.956280 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:09.956286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:09.957121 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:09.957124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:09.957789 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:09.957799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:09.958258 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:09.958261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:09.958361 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:09.958883 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:09.958897 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:09.961014 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":98,"threads_started":1}
W20260925 22:55:09.961259 30154 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:09.955392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:47033 (local address 127.20.114.254:43439)
0925 22:55:09.955526 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:09.955531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:09.955602 (+ 71us) server_negotiation.cc:407] Connection header received
0925 22:55:09.955775 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:09.955779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:09.955867 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:09.955956 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:09.956412 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:09.957008 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:09.957916 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:09.958147 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:09.958257 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:09.961142 (+ 2885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.195:47033: BlockingRecv error: recv got EOF from 127.20.114.195:47033 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 22:55:10 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:10 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:10.602762 30157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:10.597360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.193:40705)
0925 22:55:10.597759 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:10.597776 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:10.597885 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:10.598111 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:10.598113 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:10.598286 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:10.598376 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:10.598380 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:10.599083 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:10.599085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:10.599681 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:10.599689 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:10.600086 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:10.600088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:10.600150 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:10.600721 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:10.600733 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:10.602623 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":78,"threads_started":1}
W20260925 22:55:10.602916 30158 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:10.597431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:40705 (local address 127.20.114.254:43439)
0925 22:55:10.597704 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:10.597710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:10.597850 (+ 140us) server_negotiation.cc:407] Connection header received
0925 22:55:10.597957 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:10.597959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:10.598002 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:10.598077 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:10.598466 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:10.598989 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:10.599775 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:10.599994 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:10.600067 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:10.602751 (+ 2684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.193:40705: BlockingRecv error: recv got EOF from 127.20.114.193:40705 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":73,"threads_started":1}
Sep 25 22:55:10 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:10 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:10.780097 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:10.774903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.194:39813)
0925 22:55:10.775176 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:10.775189 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:10.775292 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:10.775517 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:10.775519 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:10.775691 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:10.775781 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:10.775786 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:10.776491 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:10.776493 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:10.777063 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:10.777071 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:10.777552 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:10.777554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:10.777610 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:10.778069 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:10.778079 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:10.779939 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":112,"threads_started":1}
W20260925 22:55:10.780212 30158 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:10.775018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:39813 (local address 127.20.114.254:43439)
0925 22:55:10.775164 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:10.775168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:10.775267 (+ 99us) server_negotiation.cc:407] Connection header received
0925 22:55:10.775373 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:10.775375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:10.775416 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:10.775476 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:10.775879 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:10.776403 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:10.777229 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:10.777455 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:10.777539 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:10.780091 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.194:39813: BlockingRecv error: recv got EOF from 127.20.114.194:39813 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 22:55:10 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:10 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:10.967519 30160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:10.962027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.195:34535)
0925 22:55:10.962290 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:10.962303 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:10.962397 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:10.962623 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:10.962625 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:10.962811 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:10.962903 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:10.962909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:10.963598 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:10.963600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:10.964145 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:10.964152 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:10.964658 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:10.964661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:10.964766 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:10.965451 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:10.965467 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:10.967384 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":99,"threads_started":1}
W20260925 22:55:10.967677 30158 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:10.962087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:34535 (local address 127.20.114.254:43439)
0925 22:55:10.962247 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:10.962251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:10.962369 (+ 118us) server_negotiation.cc:407] Connection header received
0925 22:55:10.962476 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:10.962478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:10.962524 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:10.962592 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:10.962999 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:10.963512 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:10.964268 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:10.964532 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:10.964676 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:10.967530 (+ 2854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.195:34535: BlockingRecv error: recv got EOF from 127.20.114.195:34535 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 22:55:11 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:11 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:11.608820 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:11.603606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.193:33325)
0925 22:55:11.603877 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:11.603891 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:11.603983 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:11.604227 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:11.604229 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:11.604410 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:11.604508 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:11.604514 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:11.605301 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:11.605304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:11.605878 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:11.605887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:11.606331 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:11.606334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:11.606394 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:11.606911 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:11.606922 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:11.608694 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":89,"threads_started":1}
W20260925 22:55:11.609051 30162 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:11.603674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:33325 (local address 127.20.114.254:43439)
0925 22:55:11.603963 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:11.603966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:11.603985 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:55:11.604051 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:11.604054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:11.604107 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:11.604191 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:11.604627 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:11.605203 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:11.605992 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:11.606231 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:11.606313 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:11.608851 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.193:33325: BlockingRecv error: recv got EOF from 127.20.114.193:33325 (error 108)
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":128,"threads_started":1}
Sep 25 22:55:11 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:11 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:11.786878 30163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:11.781136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.194:38985)
0925 22:55:11.781411 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:11.781425 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:11.781521 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:11.781756 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:11.781758 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:11.781944 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:11.782038 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:11.782044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:11.782726 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:11.782728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:11.783277 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:11.783285 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:11.783912 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:11.783916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:11.784014 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:11.784609 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:11.784621 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:11.786768 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":100,"threads_started":1}
W20260925 22:55:11.787045 30162 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:11.781192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:38985 (local address 127.20.114.254:43439)
0925 22:55:11.781334 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:11.781337 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:11.781487 (+ 150us) server_negotiation.cc:407] Connection header received
0925 22:55:11.781598 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:11.781601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:11.781644 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:11.781714 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:11.782128 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:11.782640 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:11.783425 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:11.783780 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:11.783900 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:11.786893 (+ 2993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.194:38985: BlockingRecv error: recv got EOF from 127.20.114.194:38985 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 22:55:11 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:11 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:11.974155 30164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:11.968309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.195:33067)
0925 22:55:11.968654 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:11.968676 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:11.968803 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:11.969184 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:11.969187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:11.969444 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:11.969542 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:11.969548 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:11.970451 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:11.970453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:11.971105 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:11.971114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:11.971504 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:11.971507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:11.971567 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:11.972133 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:11.972150 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:11.974038 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":113,"threads_started":1}
W20260925 22:55:11.974323 30162 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:11.968407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:33067 (local address 127.20.114.254:43439)
0925 22:55:11.968550 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:11.968556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:11.968760 (+ 204us) server_negotiation.cc:407] Connection header received
0925 22:55:11.968985 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:11.968990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:11.969065 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:11.969138 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:11.969662 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:11.970358 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:11.971203 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:11.971412 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:11.971484 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:11.974164 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.195:33067: BlockingRecv error: recv got EOF from 127.20.114.195:33067 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 22:55:12 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:12 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:12.622742 30165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:12.609817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.193:51191)
0925 22:55:12.610155 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:12.610170 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:12.610272 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:12.610579 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:12.610582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:12.610763 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:12.610857 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:12.610862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:12.611597 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:12.611600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:12.612149 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:12.612156 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:12.612668 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:12.612676 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:12.612759 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:12.620644 (+ 7885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:12.620658 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:12.622565 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":73,"threads_started":1}
W20260925 22:55:12.622822 30166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:12.609893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:51191 (local address 127.20.114.254:43439)
0925 22:55:12.610205 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:12.610209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:12.610222 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:55:12.610380 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:12.610383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:12.610426 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:12.610504 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:12.610963 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:12.611481 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:12.612251 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:12.612532 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:12.612638 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:12.622700 (+ 10062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.193:51191: BlockingRecv error: recv got EOF from 127.20.114.193:51191 (error 108)
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":92,"threads_started":1}
Sep 25 22:55:12 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:12 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:12.792701 30167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:12.787596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.194:46141)
0925 22:55:12.787889 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:12.787902 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:12.788000 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:12.788237 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:12.788240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:12.788422 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:12.788510 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:12.788515 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:12.789224 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:12.789226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:12.789778 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:12.789785 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:12.790176 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:12.790178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:12.790234 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:12.790785 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:12.790796 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:12.792591 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":97,"threads_started":1}
W20260925 22:55:12.792845 30166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:12.787710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:46141 (local address 127.20.114.254:43439)
0925 22:55:12.787892 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:12.787896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:12.787975 (+ 79us) server_negotiation.cc:407] Connection header received
0925 22:55:12.788088 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:12.788090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:12.788133 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:12.788199 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:12.788607 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:12.789138 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:12.789876 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:12.790087 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:12.790159 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:12.792715 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.194:46141: BlockingRecv error: recv got EOF from 127.20.114.194:46141 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 25 22:55:12 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:12 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:12.979832 30168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:12.974863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.195:50919)
0925 22:55:12.975120 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:12.975133 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:12.975230 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:12.975475 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:12.975477 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:12.975677 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:12.975768 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:12.975773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:12.976451 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:12.976453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:12.977031 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:12.977038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:12.977416 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:12.977418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:12.977475 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:12.977932 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:12.977943 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:12.979705 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":65,"threads_started":1}
W20260925 22:55:12.979993 30166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:12.974947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:50919 (local address 127.20.114.254:43439)
0925 22:55:12.975129 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:12.975133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:12.975201 (+ 68us) server_negotiation.cc:407] Connection header received
0925 22:55:12.975321 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:12.975323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:12.975373 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:12.975437 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:12.975861 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:12.976363 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:12.977120 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:12.977324 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:12.977396 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:12.979842 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.195:50919: BlockingRecv error: recv got EOF from 127.20.114.195:50919 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 22:55:13 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:13 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:13.629055 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:13.623511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.193:33567)
0925 22:55:13.623839 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:13.623852 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:13.623942 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:13.624181 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:13.624183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:13.624366 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:13.624458 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:13.624463 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:13.625192 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:13.625195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:13.625751 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:13.625760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:13.626176 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:13.626179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:13.626238 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:13.626761 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:13.626772 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:13.628906 (+ 2134us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":94,"threads_started":1}
W20260925 22:55:13.629336 30170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:13.623607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:33567 (local address 127.20.114.254:43439)
0925 22:55:13.623887 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:13.623890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:13.623902 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:55:13.624027 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:13.624029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:13.624072 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:13.624146 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:13.624552 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:13.625105 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:13.625863 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:13.626085 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:13.626160 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:13.629053 (+ 2893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.193:33567: BlockingRecv error: recv got EOF from 127.20.114.193:33567 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":63,"threads_started":1}
Sep 25 22:55:13 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:13 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:13.798856 30171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:13.793650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.194:40505)
0925 22:55:13.793922 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:13.793942 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:13.794063 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:13.794317 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:13.794320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:13.794514 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:13.794609 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:13.794615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:13.795316 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:13.795318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:13.795949 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:13.795957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:13.796352 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:13.796355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:13.796418 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:13.796943 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:13.796955 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:13.798730 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":98,"threads_started":1}
W20260925 22:55:13.798995 30170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:13.793666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:40505 (local address 127.20.114.254:43439)
0925 22:55:13.793836 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:13.793840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:13.794026 (+ 186us) server_negotiation.cc:407] Connection header received
0925 22:55:13.794168 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:13.794170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:13.794212 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:13.794274 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:13.794709 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:13.795223 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:13.796047 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:13.796258 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:13.796330 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:13.798867 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.194:40505: BlockingRecv error: recv got EOF from 127.20.114.194:40505 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 22:55:13 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:13 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.195@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:13.986256 30172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:13.980715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.195:40155)
0925 22:55:13.981010 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:13.981025 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:13.981131 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:13.981370 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:13.981372 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:13.981548 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:13.981635 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:13.981641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:13.982371 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:13.982373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:13.982940 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:13.982948 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:13.983348 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:13.983350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:13.983406 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:13.983988 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:13.984003 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:13.986043 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":106,"threads_started":1}
W20260925 22:55:13.986265 30170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:13.980820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:40155 (local address 127.20.114.254:43439)
0925 22:55:13.980966 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:13.980971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:13.981105 (+ 134us) server_negotiation.cc:407] Connection header received
0925 22:55:13.981213 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:13.981215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:13.981259 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:13.981337 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:13.981736 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:13.982279 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:13.983037 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:13.983254 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:13.983329 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:13.986162 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.195:40155: BlockingRecv error: recv got EOF from 127.20.114.195:40155 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 22:55:14 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:14 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.193@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:14.635490 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:14.629933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.193:39393)
0925 22:55:14.630203 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:14.630217 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:14.630299 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:14.630543 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:14.630545 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:14.630718 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:14.630806 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:14.630811 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:14.631537 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:14.631543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:14.632243 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:14.632253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:14.632676 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:14.632678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:14.632745 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:14.633351 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:14.633362 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:14.635303 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":69,"threads_started":1}
W20260925 22:55:14.635597 30174 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:14.630013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:39393 (local address 127.20.114.254:43439)
0925 22:55:14.630254 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:14.630258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:14.630271 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:55:14.630390 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:14.630393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:14.630434 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:14.630511 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:14.630904 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:14.631423 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:14.632364 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:14.632583 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:14.632658 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:14.635464 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.193:39393: BlockingRecv error: recv got EOF from 127.20.114.193:39393 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":75,"threads_started":1}
Sep 25 22:55:14 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:55:14 dist-test-slave-x1lh krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.114.194@KRBTEST.COM for oryx/127.20.114.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:55:14.805248 30175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:14.799800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43439 (local address 127.20.114.194:45327)
0925 22:55:14.800057 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:14.800070 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:14.800158 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:14.800363 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:14.800365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:14.800542 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:14.800629 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:14.800633 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:14.801361 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:14.801363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:14.801919 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:14.801927 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:14.802337 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:14.802340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:14.802398 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:14.802943 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:14.802954 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:14.805057 (+ 2103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.114.254:43439: Server oryx/127.20.114.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":96,"threads_started":1}
W20260925 22:55:14.805333 30174 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:14.799894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:45327 (local address 127.20.114.254:43439)
0925 22:55:14.800040 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:14.800044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:14.800115 (+ 71us) server_negotiation.cc:407] Connection header received
0925 22:55:14.800218 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:14.800220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:14.800261 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:14.800332 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:14.800720 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:14.801264 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:14.802025 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:14.802243 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:14.802321 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:14.805211 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.114.194:45327: BlockingRecv error: recv got EOF from 127.20.114.194:45327 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:55:14.887377 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 29699
I20260925 22:55:14.893164 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 29851
I20260925 22:55:14.898401 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 29983
I20260925 22:55:14.903694 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 29629
2026-09-25T22:55:14Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17939 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T22:55:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:55:14Z Disabled control of system clock
I20260925 22:55:14.934237 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:36123
--webserver_interface=127.20.114.254
--webserver_port=0
--builtin_ntp_servers=127.20.114.212:46389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:36123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 22:55:15.040081 30179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:55:15.040382 30179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:55:15.040438 30179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:55:15.044328 30179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:55:15.044418 30179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:55:15.044449 30179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:55:15.044488 30179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:55:15.044523 30179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:55:15.049170 30179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:46389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:36123
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:36123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:55:15.050226 30179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:55:15.051551 30179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:55:15.060245 30184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:55:15.060333 30179 server_base.cc:1034] running on GCE node
W20260925 22:55:15.060278 30185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:55:15.060277 30187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:55:15.061028 30179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:55:15.062024 30179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:55:15.063194 30179 hybrid_clock.cc:648] HybridClock initialized: now 1790376915063178 us; error 25 us; skew 500 ppm
I20260925 22:55:15.065248 30179 webserver.cc:460] Webserver started at http://127.20.114.254:36419/ using document root <none> and password file <none>
I20260925 22:55:15.065825 30179 fs_manager.cc:359] Metadata directory not provided
I20260925 22:55:15.065884 30179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:55:15.066141 30179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:55:15.068112 30179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "38fcb77c152842bf83056fd853d0ecd1"
format_stamp: "Formatted at 2026-09-25 22:55:15 on dist-test-slave-x1lh"
server_key: "b7ae91b40cbad24898ca767a096a4464"
server_key_iv: "5828f90a212c7b2e5ea4066e47f10d73"
server_key_version: "encryptionkey@0"
I20260925 22:55:15.068679 30179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "38fcb77c152842bf83056fd853d0ecd1"
format_stamp: "Formatted at 2026-09-25 22:55:15 on dist-test-slave-x1lh"
server_key: "b7ae91b40cbad24898ca767a096a4464"
server_key_iv: "5828f90a212c7b2e5ea4066e47f10d73"
server_key_version: "encryptionkey@0"
I20260925 22:55:15.072309 30179 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:55:15.074674 30193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:15.075838 30179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:55:15.075985 30179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "38fcb77c152842bf83056fd853d0ecd1"
format_stamp: "Formatted at 2026-09-25 22:55:15 on dist-test-slave-x1lh"
server_key: "b7ae91b40cbad24898ca767a096a4464"
server_key_iv: "5828f90a212c7b2e5ea4066e47f10d73"
server_key_version: "encryptionkey@0"
I20260925 22:55:15.076115 30179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:55:15.086253 30179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:55:15.086894 30179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:55:15.087080 30179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:55:15.093649 30179 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:36123
I20260925 22:55:15.093669 30245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:36123 every 8 connection(s)
I20260925 22:55:15.094590 30179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:55:15.097710 30246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:55:15.099409 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 30179
I20260925 22:55:15.099517 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:55:15.099797 20939 external_mini_cluster.cc:1428] Setting key 9d84bb9e2690f862b2e05c5023406e4e
I20260925 22:55:15.104102 30246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38fcb77c152842bf83056fd853d0ecd1: Bootstrap starting.
I20260925 22:55:15.106281 30246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38fcb77c152842bf83056fd853d0ecd1: Neither blocks nor log segments found. Creating new log.
I20260925 22:55:15.107136 30246 log.cc:824] T 00000000000000000000000000000000 P 38fcb77c152842bf83056fd853d0ecd1: Log is configured to *not* fsync() on all Append() calls
I20260925 22:55:15.109108 30249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:15.101277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57086 (local address 127.20.114.254:36123)
0925 22:55:15.101887 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:15.101902 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:15.101936 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:55:15.102584 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:15.102600 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:15.102981 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:15.103319 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:15.103732 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:15.104709 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:15.105646 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:15.106118 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:15.106285 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:15.107180 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:55:15.107185 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:55:15.107198 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:55:15.107697 (+ 499us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:55:15.107769 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:55:15.107797 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:55:15.107909 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:55:15.108069 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 22:55:15.108362 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":114,"threads_started":1}
I20260925 22:55:15.109122 30246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38fcb77c152842bf83056fd853d0ecd1: No bootstrap required, opened a new log
I20260925 22:55:15.112931 30246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38fcb77c152842bf83056fd853d0ecd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fcb77c152842bf83056fd853d0ecd1" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36123 } }
I20260925 22:55:15.113165 30246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38fcb77c152842bf83056fd853d0ecd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:55:15.113240 30246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38fcb77c152842bf83056fd853d0ecd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38fcb77c152842bf83056fd853d0ecd1, State: Initialized, Role: FOLLOWER
I20260925 22:55:15.113687 30246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38fcb77c152842bf83056fd853d0ecd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fcb77c152842bf83056fd853d0ecd1" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36123 } }
I20260925 22:55:15.113837 30246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38fcb77c152842bf83056fd853d0ecd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:55:15.113921 30246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38fcb77c152842bf83056fd853d0ecd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:55:15.114017 30246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38fcb77c152842bf83056fd853d0ecd1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:55:15.114868 30246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38fcb77c152842bf83056fd853d0ecd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fcb77c152842bf83056fd853d0ecd1" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36123 } }
I20260925 22:55:15.115187 30246 leader_election.cc:302] T 00000000000000000000000000000000 P 38fcb77c152842bf83056fd853d0ecd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38fcb77c152842bf83056fd853d0ecd1; no voters:
I20260925 22:55:15.115486 30246 leader_election.cc:288] T 00000000000000000000000000000000 P 38fcb77c152842bf83056fd853d0ecd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:55:15.115650 30251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38fcb77c152842bf83056fd853d0ecd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:55:15.115892 30251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38fcb77c152842bf83056fd853d0ecd1 [term 1 LEADER]: Becoming Leader. State: Replica: 38fcb77c152842bf83056fd853d0ecd1, State: Running, Role: LEADER
I20260925 22:55:15.116217 30251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38fcb77c152842bf83056fd853d0ecd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fcb77c152842bf83056fd853d0ecd1" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36123 } }
I20260925 22:55:15.116659 30246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38fcb77c152842bf83056fd853d0ecd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:55:15.117789 30253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38fcb77c152842bf83056fd853d0ecd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38fcb77c152842bf83056fd853d0ecd1. Latest consensus state: current_term: 1 leader_uuid: "38fcb77c152842bf83056fd853d0ecd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fcb77c152842bf83056fd853d0ecd1" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36123 } } }
I20260925 22:55:15.117916 30253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38fcb77c152842bf83056fd853d0ecd1 [sys.catalog]: This master's current role is: LEADER
I20260925 22:55:15.118232 30252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38fcb77c152842bf83056fd853d0ecd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38fcb77c152842bf83056fd853d0ecd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fcb77c152842bf83056fd853d0ecd1" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36123 } } }
I20260925 22:55:15.118297 30260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:55:15.118333 30252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38fcb77c152842bf83056fd853d0ecd1 [sys.catalog]: This master's current role is: LEADER
I20260925 22:55:15.121368 30260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:55:15.126463 30260 catalog_manager.cc:1347] Generated new cluster ID: d71b71f300b6436a879275fcaaeec0b8
I20260925 22:55:15.126530 30260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:55:15.140900 30260 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:55:15.142073 30260 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:55:15.152519 30260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38fcb77c152842bf83056fd853d0ecd1: Generated new TSK 0
I20260925 22:55:15.153373 30260 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:55:15.165199 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:36123
--builtin_ntp_servers=127.20.114.212:46389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 22:55:15.273533 30270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:55:15.273787 30270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:55:15.273824 30270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:55:15.277814 30270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:55:15.277865 30270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:55:15.277938 30270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:55:15.284262 30270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:46389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:36123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:55:15.285444 30270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:55:15.286808 30270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:55:15.296307 30276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:55:15.296286 30275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:55:15.296286 30278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:55:15.296562 30270 server_base.cc:1034] running on GCE node
I20260925 22:55:15.296998 30270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:55:15.297642 30270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:55:15.298795 30270 hybrid_clock.cc:648] HybridClock initialized: now 1790376915298765 us; error 48 us; skew 500 ppm
I20260925 22:55:15.300731 30270 webserver.cc:460] Webserver started at http://127.20.114.193:33353/ using document root <none> and password file <none>
I20260925 22:55:15.301343 30270 fs_manager.cc:359] Metadata directory not provided
I20260925 22:55:15.301404 30270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:55:15.301635 30270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:55:15.303646 30270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "931dd0dae4004329bd6412f2b8c5a295"
format_stamp: "Formatted at 2026-09-25 22:55:15 on dist-test-slave-x1lh"
server_key: "7dcc4c7d28e7d7ef330841eed3a15594"
server_key_iv: "ca954a04aa010d528158f58a1ba263da"
server_key_version: "encryptionkey@0"
I20260925 22:55:15.304207 30270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "931dd0dae4004329bd6412f2b8c5a295"
format_stamp: "Formatted at 2026-09-25 22:55:15 on dist-test-slave-x1lh"
server_key: "7dcc4c7d28e7d7ef330841eed3a15594"
server_key_iv: "ca954a04aa010d528158f58a1ba263da"
server_key_version: "encryptionkey@0"
I20260925 22:55:15.307868 30270 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:55:15.310357 30284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:15.311597 30270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:55:15.311731 30270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "931dd0dae4004329bd6412f2b8c5a295"
format_stamp: "Formatted at 2026-09-25 22:55:15 on dist-test-slave-x1lh"
server_key: "7dcc4c7d28e7d7ef330841eed3a15594"
server_key_iv: "ca954a04aa010d528158f58a1ba263da"
server_key_version: "encryptionkey@0"
I20260925 22:55:15.311856 30270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:55:15.334834 30270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:55:15.335575 30270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:55:15.335785 30270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:55:15.336385 30270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:55:15.337348 30270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:55:15.337419 30270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:15.337488 30270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:55:15.337522 30270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:15.346397 30270 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:33351
I20260925 22:55:15.346423 30397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:33351 every 8 connection(s)
I20260925 22:55:15.347322 30270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:55:15.351329 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 30270
I20260925 22:55:15.351456 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:55:15.351817 20939 external_mini_cluster.cc:1428] Setting key 57e6665702cdfdc519226bc4f98b7fbe
I20260925 22:55:15.353960 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:36123
--builtin_ntp_servers=127.20.114.212:46389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 22:55:15.358397 30249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:15.349322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:42951 (local address 127.20.114.254:36123)
0925 22:55:15.349461 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:15.349464 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:15.350570 (+ 1106us) server_negotiation.cc:407] Connection header received
0925 22:55:15.352307 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:15.352312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:15.352379 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:15.352484 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:15.353656 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:15.354274 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:15.355844 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:15.356228 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:15.356368 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:15.357737 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:55:15.357744 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:55:15.357748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:55:15.357767 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:55:15.357793 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:55:15.357806 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:55:15.357889 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:55:15.358056 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 22:55:15.358179 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:55:15.359131 30400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:15.349785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:36123 (local address 127.20.114.193:42951)
0925 22:55:15.350420 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:15.350460 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:15.352032 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:15.352669 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:15.352678 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:15.352786 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:15.353497 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:15.353518 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:15.354396 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:15.354403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:15.355714 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:15.355723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:15.356409 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:15.356414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:15.356594 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:15.357406 (+ 812us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:55:15.357408 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:15.357464 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:55:15.357549 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:55:15.357904 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:55:15.357910 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:55:15.357915 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:55:15.358017 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 22:55:15.358268 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":107,"threads_started":1}
I20260925 22:55:15.360437 30398 heartbeater.cc:347] Connected to a master server at 127.20.114.254:36123
I20260925 22:55:15.360687 30398 heartbeater.cc:464] Registering TS with master...
I20260925 22:55:15.361454 30398 heartbeater.cc:511] Master 127.20.114.254:36123 requested a full tablet report, sending...
I20260925 22:55:15.363349 30210 ts_manager.cc:194] Registered new tserver with Master: 931dd0dae4004329bd6412f2b8c5a295 (127.20.114.193:33351)
I20260925 22:55:15.364710 30210 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.114.193:42951
W20260925 22:55:15.471024 30401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:55:15.471268 30401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:55:15.471304 30401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:55:15.475901 30401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:55:15.475960 30401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:55:15.476011 30401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:55:15.480759 30401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:46389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:36123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:55:15.481760 30401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:55:15.483057 30401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:55:15.492301 30407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:55:15.492327 30409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:55:15.492352 30406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:55:15.492548 30401 server_base.cc:1034] running on GCE node
I20260925 22:55:15.493163 30401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:55:15.493775 30401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:55:15.494968 30401 hybrid_clock.cc:648] HybridClock initialized: now 1790376915494955 us; error 31 us; skew 500 ppm
I20260925 22:55:15.496735 30401 webserver.cc:460] Webserver started at http://127.20.114.194:33887/ using document root <none> and password file <none>
I20260925 22:55:15.497346 30401 fs_manager.cc:359] Metadata directory not provided
I20260925 22:55:15.497438 30401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:55:15.497660 30401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:55:15.499606 30401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "8441da1a45f743efbf1d395cafe7d5ca"
format_stamp: "Formatted at 2026-09-25 22:55:15 on dist-test-slave-x1lh"
server_key: "5bc16b27f1a237e15308527156dad890"
server_key_iv: "c6aa16fd5288f0a39896095cae1de1c6"
server_key_version: "encryptionkey@0"
I20260925 22:55:15.500185 30401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "8441da1a45f743efbf1d395cafe7d5ca"
format_stamp: "Formatted at 2026-09-25 22:55:15 on dist-test-slave-x1lh"
server_key: "5bc16b27f1a237e15308527156dad890"
server_key_iv: "c6aa16fd5288f0a39896095cae1de1c6"
server_key_version: "encryptionkey@0"
I20260925 22:55:15.503687 30401 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:55:15.506101 30415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:15.507253 30401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:55:15.507395 30401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "8441da1a45f743efbf1d395cafe7d5ca"
format_stamp: "Formatted at 2026-09-25 22:55:15 on dist-test-slave-x1lh"
server_key: "5bc16b27f1a237e15308527156dad890"
server_key_iv: "c6aa16fd5288f0a39896095cae1de1c6"
server_key_version: "encryptionkey@0"
I20260925 22:55:15.507506 30401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:55:15.544014 30401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:55:15.544723 30401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:55:15.544946 30401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:55:15.545539 30401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:55:15.546487 30401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:55:15.546558 30401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:15.546629 30401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:55:15.546672 30401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:15.555053 30401 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:45025
I20260925 22:55:15.555075 30528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:45025 every 8 connection(s)
I20260925 22:55:15.555938 30401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:55:15.560912 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 30401
I20260925 22:55:15.561030 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:55:15.561309 20939 external_mini_cluster.cc:1428] Setting key 71eb410ddb881dcb7922785b7cf0f2ba
I20260925 22:55:15.563169 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:36123
--builtin_ntp_servers=127.20.114.212:46389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 22:55:15.565478 30249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:15.557863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:44129 (local address 127.20.114.254:36123)
0925 22:55:15.558006 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:15.558009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:15.559081 (+ 1072us) server_negotiation.cc:407] Connection header received
0925 22:55:15.560457 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:15.560461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:15.560516 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:15.560620 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:15.561671 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:15.562298 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:15.563498 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:15.563716 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:15.563826 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:15.565027 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:55:15.565030 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:55:15.565032 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:55:15.565044 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:55:15.565064 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:55:15.565072 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:55:15.565135 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 22:55:15.565316 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 22:55:15.565369 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:55:15.566349 30531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:15.558297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:36123 (local address 127.20.114.194:44129)
0925 22:55:15.558937 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:15.558976 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:15.560255 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:15.560806 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:15.560813 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:15.560952 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:15.561540 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:15.561554 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:15.562421 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:15.562424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:15.563384 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:15.563393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:15.563875 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:15.563879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:15.564041 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:15.564699 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:55:15.564702 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:15.564759 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:55:15.564842 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:55:15.565161 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:55:15.565170 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:55:15.565175 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:55:15.565318 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 22:55:15.565620 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":106,"threads_started":1}
I20260925 22:55:15.567461 30529 heartbeater.cc:347] Connected to a master server at 127.20.114.254:36123
I20260925 22:55:15.567713 30529 heartbeater.cc:464] Registering TS with master...
I20260925 22:55:15.568209 30529 heartbeater.cc:511] Master 127.20.114.254:36123 requested a full tablet report, sending...
I20260925 22:55:15.569262 30210 ts_manager.cc:194] Registered new tserver with Master: 8441da1a45f743efbf1d395cafe7d5ca (127.20.114.194:45025)
I20260925 22:55:15.569852 30210 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.114.194:44129
W20260925 22:55:15.674793 30532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:55:15.675061 30532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:55:15.675123 30532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:55:15.679111 30532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:55:15.679188 30532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:55:15.679273 30532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:55:15.683987 30532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:46389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:36123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:55:15.685106 30532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:55:15.686470 30532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:55:15.695864 30538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:55:15.695864 30537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:55:15.695887 30540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:55:15.696084 30532 server_base.cc:1034] running on GCE node
I20260925 22:55:15.696584 30532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:55:15.697274 30532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:55:15.698429 30532 hybrid_clock.cc:648] HybridClock initialized: now 1790376915698416 us; error 23 us; skew 500 ppm
I20260925 22:55:15.700168 30532 webserver.cc:460] Webserver started at http://127.20.114.195:44685/ using document root <none> and password file <none>
I20260925 22:55:15.700732 30532 fs_manager.cc:359] Metadata directory not provided
I20260925 22:55:15.700796 30532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:55:15.701056 30532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:55:15.703064 30532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "0cbbc14512074d6ea02a8b4bb34e85bb"
format_stamp: "Formatted at 2026-09-25 22:55:15 on dist-test-slave-x1lh"
server_key: "0cdf65d9a75c5c5fbfc2085e66af8504"
server_key_iv: "25e2f9acdb61657d30dfe360eba32c3b"
server_key_version: "encryptionkey@0"
I20260925 22:55:15.703620 30532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "0cbbc14512074d6ea02a8b4bb34e85bb"
format_stamp: "Formatted at 2026-09-25 22:55:15 on dist-test-slave-x1lh"
server_key: "0cdf65d9a75c5c5fbfc2085e66af8504"
server_key_iv: "25e2f9acdb61657d30dfe360eba32c3b"
server_key_version: "encryptionkey@0"
I20260925 22:55:15.707147 30532 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 22:55:15.709733 30546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:15.710843 30532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:55:15.710973 30532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "0cbbc14512074d6ea02a8b4bb34e85bb"
format_stamp: "Formatted at 2026-09-25 22:55:15 on dist-test-slave-x1lh"
server_key: "0cdf65d9a75c5c5fbfc2085e66af8504"
server_key_iv: "25e2f9acdb61657d30dfe360eba32c3b"
server_key_version: "encryptionkey@0"
I20260925 22:55:15.711082 30532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:55:15.744735 30532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:55:15.745502 30532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:55:15.745702 30532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:55:15.746312 30532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:55:15.747228 30532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:55:15.747306 30532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:15.747377 30532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:55:15.747429 30532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:15.756088 30532 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:39979
I20260925 22:55:15.756124 30659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:39979 every 8 connection(s)
I20260925 22:55:15.756999 30532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:55:15.760519 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 30532
I20260925 22:55:15.760620 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:55:15.760924 20939 external_mini_cluster.cc:1428] Setting key 26f54ff38d76767595e822744c85af2e
I20260925 22:55:15.766402 30249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:15.758852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:47395 (local address 127.20.114.254:36123)
0925 22:55:15.758983 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:15.758987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:15.760029 (+ 1042us) server_negotiation.cc:407] Connection header received
0925 22:55:15.761809 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:15.761812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:15.761869 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:15.761951 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:15.762833 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:15.763386 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:15.764555 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:15.764787 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:15.764925 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:15.765914 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:55:15.765917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:55:15.765920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:55:15.765932 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:55:15.765960 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:55:15.765969 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:55:15.766048 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 22:55:15.766199 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 22:55:15.766257 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:55:15.767251 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:15.759246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:36123 (local address 127.20.114.195:47395)
0925 22:55:15.759885 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:15.759924 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:15.761590 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:15.762135 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:15.762142 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:15.762224 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:15.762696 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:15.762709 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:15.763521 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:15.763525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:15.764390 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:15.764400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:15.764961 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:15.764964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:15.765126 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:15.765654 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:55:15.765656 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:15.765694 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:55:15.765758 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:55:15.766049 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:55:15.766060 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:55:15.766068 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 22:55:15.766198 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 22:55:15.766471 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":102,"threads_started":1}
I20260925 22:55:15.768249 30660 heartbeater.cc:347] Connected to a master server at 127.20.114.254:36123
I20260925 22:55:15.768469 30660 heartbeater.cc:464] Registering TS with master...
I20260925 22:55:15.769017 30660 heartbeater.cc:511] Master 127.20.114.254:36123 requested a full tablet report, sending...
I20260925 22:55:15.769987 30210 ts_manager.cc:194] Registered new tserver with Master: 0cbbc14512074d6ea02a8b4bb34e85bb (127.20.114.195:39979)
I20260925 22:55:15.770550 30210 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.114.195:47395
I20260925 22:55:15.774464 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 22:55:15.777009 30249 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:15.776185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57092 (local address 127.20.114.254:36123)
0925 22:55:15.776306 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:15.776310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:15.776462 (+ 152us) server_negotiation.cc:407] Connection header received
0925 22:55:15.776558 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:15.776560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:15.776593 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:15.776654 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:15.776846 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57092: BlockingRecv error: recv got EOF from 127.0.0.1:57092 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:55:15.777276 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 30270
I20260925 22:55:15.783716 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 30401
I20260925 22:55:15.788827 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 30532
I20260925 22:55:15.793915 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 30179
2026-09-25T22:55:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (888 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T22:55:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:55:15Z Disabled control of system clock
I20260925 22:55:15.822499 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:40037
--webserver_interface=127.20.114.254
--webserver_port=0
--builtin_ntp_servers=127.20.114.212:44245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:40037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 22:55:15.933410 30671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:55:15.933688 30671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:55:15.933725 30671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:55:15.937667 30671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:55:15.937722 30671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:55:15.937739 30671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:55:15.937755 30671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:55:15.937770 30671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:55:15.942559 30671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:44245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:40037
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:40037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:55:15.943599 30671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:55:15.945011 30671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:55:15.953405 30677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:55:15.953392 30676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:55:15.953528 30671 server_base.cc:1034] running on GCE node
W20260925 22:55:15.953387 30679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:55:15.954226 30671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:55:15.955394 30671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:55:15.956583 30671 hybrid_clock.cc:648] HybridClock initialized: now 1790376915956568 us; error 30 us; skew 500 ppm
I20260925 22:55:15.958510 30671 webserver.cc:460] Webserver started at http://127.20.114.254:37239/ using document root <none> and password file <none>
I20260925 22:55:15.959110 30671 fs_manager.cc:359] Metadata directory not provided
I20260925 22:55:15.959197 30671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:55:15.959427 30671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:55:15.961565 30671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "4ed33fe922024d14a392362c156d6efe"
format_stamp: "Formatted at 2026-09-25 22:55:15 on dist-test-slave-x1lh"
server_key: "9318a8196e535ace9f6b55b1f0617025"
server_key_iv: "8b50ccf7b3e231cb0fbb125ab39d2d4a"
server_key_version: "encryptionkey@0"
I20260925 22:55:15.962131 30671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "4ed33fe922024d14a392362c156d6efe"
format_stamp: "Formatted at 2026-09-25 22:55:15 on dist-test-slave-x1lh"
server_key: "9318a8196e535ace9f6b55b1f0617025"
server_key_iv: "8b50ccf7b3e231cb0fbb125ab39d2d4a"
server_key_version: "encryptionkey@0"
I20260925 22:55:15.965813 30671 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:55:15.968147 30685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:15.969354 30671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:55:15.969493 30671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "4ed33fe922024d14a392362c156d6efe"
format_stamp: "Formatted at 2026-09-25 22:55:15 on dist-test-slave-x1lh"
server_key: "9318a8196e535ace9f6b55b1f0617025"
server_key_iv: "8b50ccf7b3e231cb0fbb125ab39d2d4a"
server_key_version: "encryptionkey@0"
I20260925 22:55:15.969619 30671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:55:15.974259 30671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:55:15.974849 30671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:55:15.975029 30671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:55:15.981679 30671 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:40037
I20260925 22:55:15.981724 30737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:40037 every 8 connection(s)
I20260925 22:55:15.982622 30671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:55:15.985780 30738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:55:15.987581 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 30671
I20260925 22:55:15.987717 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:55:15.987989 20939 external_mini_cluster.cc:1428] Setting key b9328233447970e4b5417f9bda4b5a0f
I20260925 22:55:15.991679 30738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ed33fe922024d14a392362c156d6efe: Bootstrap starting.
I20260925 22:55:15.993582 30741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:15.989439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47288 (local address 127.20.114.254:40037)
0925 22:55:15.990080 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:15.990094 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:15.990128 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:55:15.990865 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:15.990884 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:15.991223 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:15.991564 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:15.992302 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:55:15.992307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:55:15.992328 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:55:15.992337 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:55:15.992385 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:55:15.992401 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:55:15.992549 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 22:55:15.992620 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 22:55:15.992855 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":141,"threads_started":1}
I20260925 22:55:15.994134 30738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ed33fe922024d14a392362c156d6efe: Neither blocks nor log segments found. Creating new log.
I20260925 22:55:15.994984 30738 log.cc:824] T 00000000000000000000000000000000 P 4ed33fe922024d14a392362c156d6efe: Log is configured to *not* fsync() on all Append() calls
I20260925 22:55:15.996990 30738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ed33fe922024d14a392362c156d6efe: No bootstrap required, opened a new log
I20260925 22:55:16.000576 30738 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ed33fe922024d14a392362c156d6efe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed33fe922024d14a392362c156d6efe" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40037 } }
I20260925 22:55:16.000792 30738 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ed33fe922024d14a392362c156d6efe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:55:16.000825 30738 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ed33fe922024d14a392362c156d6efe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed33fe922024d14a392362c156d6efe, State: Initialized, Role: FOLLOWER
I20260925 22:55:16.001305 30738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ed33fe922024d14a392362c156d6efe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed33fe922024d14a392362c156d6efe" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40037 } }
I20260925 22:55:16.001432 30738 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ed33fe922024d14a392362c156d6efe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:55:16.001479 30738 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ed33fe922024d14a392362c156d6efe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:55:16.001550 30738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ed33fe922024d14a392362c156d6efe [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:55:16.002394 30738 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ed33fe922024d14a392362c156d6efe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed33fe922024d14a392362c156d6efe" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40037 } }
I20260925 22:55:16.002678 30738 leader_election.cc:302] T 00000000000000000000000000000000 P 4ed33fe922024d14a392362c156d6efe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ed33fe922024d14a392362c156d6efe; no voters:
I20260925 22:55:16.002974 30738 leader_election.cc:288] T 00000000000000000000000000000000 P 4ed33fe922024d14a392362c156d6efe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:55:16.003082 30743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ed33fe922024d14a392362c156d6efe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:55:16.003331 30743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ed33fe922024d14a392362c156d6efe [term 1 LEADER]: Becoming Leader. State: Replica: 4ed33fe922024d14a392362c156d6efe, State: Running, Role: LEADER
I20260925 22:55:16.003618 30743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ed33fe922024d14a392362c156d6efe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed33fe922024d14a392362c156d6efe" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40037 } }
I20260925 22:55:16.004315 30738 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ed33fe922024d14a392362c156d6efe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:55:16.005155 30745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ed33fe922024d14a392362c156d6efe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ed33fe922024d14a392362c156d6efe. Latest consensus state: current_term: 1 leader_uuid: "4ed33fe922024d14a392362c156d6efe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed33fe922024d14a392362c156d6efe" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40037 } } }
I20260925 22:55:16.005280 30745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ed33fe922024d14a392362c156d6efe [sys.catalog]: This master's current role is: LEADER
I20260925 22:55:16.005577 30744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ed33fe922024d14a392362c156d6efe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ed33fe922024d14a392362c156d6efe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed33fe922024d14a392362c156d6efe" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40037 } } }
I20260925 22:55:16.005673 30744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ed33fe922024d14a392362c156d6efe [sys.catalog]: This master's current role is: LEADER
I20260925 22:55:16.006047 30752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:55:16.009096 30752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:55:16.014860 30752 catalog_manager.cc:1347] Generated new cluster ID: 0f4c923129ba46e8bdf083ee5ab9cf8d
I20260925 22:55:16.014931 30752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:55:16.038147 30752 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:55:16.038498 30752 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:55:16.044798 30752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ed33fe922024d14a392362c156d6efe: Generated new TSK 0
I20260925 22:55:16.045660 30752 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:55:16.054908 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:40037
--builtin_ntp_servers=127.20.114.212:44245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 22:55:16.165395 30762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:55:16.165668 30762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:55:16.165709 30762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:55:16.169723 30762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:55:16.169785 30762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:55:16.169843 30762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:55:16.174903 30762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:44245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:40037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:55:16.175979 30762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:55:16.177368 30762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:55:16.187069 30770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:55:16.187068 30768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:55:16.187064 30767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:55:16.187345 30762 server_base.cc:1034] running on GCE node
I20260925 22:55:16.187839 30762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:55:16.188540 30762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:55:16.189724 30762 hybrid_clock.cc:648] HybridClock initialized: now 1790376916189708 us; error 30 us; skew 500 ppm
I20260925 22:55:16.191635 30762 webserver.cc:460] Webserver started at http://127.20.114.193:42917/ using document root <none> and password file <none>
I20260925 22:55:16.192251 30762 fs_manager.cc:359] Metadata directory not provided
I20260925 22:55:16.192348 30762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:55:16.192593 30762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:55:16.194811 30762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "3717b8aec88549b88a334a47f4b2788f"
format_stamp: "Formatted at 2026-09-25 22:55:16 on dist-test-slave-x1lh"
server_key: "1a5369e7aea6cec541998285af5516e0"
server_key_iv: "bc7ba63ec8a72a3c947170dac7ca26d2"
server_key_version: "encryptionkey@0"
I20260925 22:55:16.195420 30762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "3717b8aec88549b88a334a47f4b2788f"
format_stamp: "Formatted at 2026-09-25 22:55:16 on dist-test-slave-x1lh"
server_key: "1a5369e7aea6cec541998285af5516e0"
server_key_iv: "bc7ba63ec8a72a3c947170dac7ca26d2"
server_key_version: "encryptionkey@0"
I20260925 22:55:16.199239 30762 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:55:16.201896 30776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:16.203119 30762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:55:16.203261 30762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "3717b8aec88549b88a334a47f4b2788f"
format_stamp: "Formatted at 2026-09-25 22:55:16 on dist-test-slave-x1lh"
server_key: "1a5369e7aea6cec541998285af5516e0"
server_key_iv: "bc7ba63ec8a72a3c947170dac7ca26d2"
server_key_version: "encryptionkey@0"
I20260925 22:55:16.203392 30762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:55:16.208289 30762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:55:16.208997 30762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:55:16.209187 30762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:55:16.209767 30762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:55:16.210704 30762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:55:16.210777 30762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:16.210857 30762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:55:16.210901 30762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:16.219424 30762 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:44803
I20260925 22:55:16.219442 30889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:44803 every 8 connection(s)
I20260925 22:55:16.220337 30762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:55:16.220970 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 30762
I20260925 22:55:16.221078 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:55:16.221386 20939 external_mini_cluster.cc:1428] Setting key 307943cd848ce4ef6bb3a8af857f3cca
I20260925 22:55:16.223659 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:40037
--builtin_ntp_servers=127.20.114.212:44245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 22:55:16.227686 30741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:16.222310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:36051 (local address 127.20.114.254:40037)
0925 22:55:16.222455 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:16.222460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:16.223822 (+ 1362us) server_negotiation.cc:407] Connection header received
0925 22:55:16.225255 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:16.225258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:16.225302 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:16.225371 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:16.227096 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:55:16.227103 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:55:16.227108 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:55:16.227126 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:55:16.227156 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:55:16.227170 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:55:16.227277 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:55:16.227441 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 22:55:16.227498 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:55:16.228453 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:16.222832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:40037 (local address 127.20.114.193:36051)
0925 22:55:16.223589 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:16.223630 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:16.225054 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:16.225534 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:16.225541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:16.225631 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:16.226822 (+ 1191us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:55:16.226824 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:16.226864 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:55:16.226933 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:55:16.227282 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:55:16.227290 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:55:16.227296 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:55:16.227441 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 22:55:16.227631 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":138,"threads_started":1}
I20260925 22:55:16.229744 30890 heartbeater.cc:347] Connected to a master server at 127.20.114.254:40037
I20260925 22:55:16.229983 30890 heartbeater.cc:464] Registering TS with master...
I20260925 22:55:16.230386 30890 heartbeater.cc:511] Master 127.20.114.254:40037 requested a full tablet report, sending...
I20260925 22:55:16.232043 30702 ts_manager.cc:194] Registered new tserver with Master: 3717b8aec88549b88a334a47f4b2788f (127.20.114.193:44803)
W20260925 22:55:16.337148 30893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:55:16.337416 30893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:55:16.337455 30893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:55:16.341347 30893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:55:16.341395 30893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:55:16.341449 30893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:55:16.346045 30893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:44245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:40037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:55:16.347077 30893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:55:16.348445 30893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:55:16.357779 30901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:55:16.357786 30898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:55:16.357859 30899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:55:16.358160 30893 server_base.cc:1034] running on GCE node
I20260925 22:55:16.358523 30893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:55:16.359195 30893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:55:16.360378 30893 hybrid_clock.cc:648] HybridClock initialized: now 1790376916360350 us; error 38 us; skew 500 ppm
I20260925 22:55:16.362248 30893 webserver.cc:460] Webserver started at http://127.20.114.194:39251/ using document root <none> and password file <none>
I20260925 22:55:16.362877 30893 fs_manager.cc:359] Metadata directory not provided
I20260925 22:55:16.362970 30893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:55:16.363207 30893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:55:16.365345 30893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "b622dc6f413340dbbac6ed4e3e9177b4"
format_stamp: "Formatted at 2026-09-25 22:55:16 on dist-test-slave-x1lh"
server_key: "bf1ed2d2a222b11001d0cb365ca2886f"
server_key_iv: "cc0db348997cf84b583d86bbbc111982"
server_key_version: "encryptionkey@0"
I20260925 22:55:16.365931 30893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "b622dc6f413340dbbac6ed4e3e9177b4"
format_stamp: "Formatted at 2026-09-25 22:55:16 on dist-test-slave-x1lh"
server_key: "bf1ed2d2a222b11001d0cb365ca2886f"
server_key_iv: "cc0db348997cf84b583d86bbbc111982"
server_key_version: "encryptionkey@0"
I20260925 22:55:16.369583 30893 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:55:16.372061 30907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:16.373350 30893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 22:55:16.373478 30893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "b622dc6f413340dbbac6ed4e3e9177b4"
format_stamp: "Formatted at 2026-09-25 22:55:16 on dist-test-slave-x1lh"
server_key: "bf1ed2d2a222b11001d0cb365ca2886f"
server_key_iv: "cc0db348997cf84b583d86bbbc111982"
server_key_version: "encryptionkey@0"
I20260925 22:55:16.373592 30893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:55:16.378286 30893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:55:16.378913 30893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:55:16.379117 30893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:55:16.379676 30893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:55:16.380594 30893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:55:16.380666 30893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:16.380730 30893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:55:16.380779 30893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:16.389269 30893 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:35803
I20260925 22:55:16.389295 31020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:35803 every 8 connection(s)
I20260925 22:55:16.390188 30893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:55:16.395979 30741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:16.391906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:46753 (local address 127.20.114.254:40037)
0925 22:55:16.392063 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:16.392068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:16.393169 (+ 1101us) server_negotiation.cc:407] Connection header received
0925 22:55:16.394396 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:16.394398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:16.394436 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:16.394513 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:16.395619 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:55:16.395622 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:55:16.395623 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:55:16.395632 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:55:16.395647 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:55:16.395654 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:55:16.395727 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 22:55:16.395832 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 22:55:16.395872 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:55:16.396814 31023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:16.392368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:40037 (local address 127.20.114.194:46753)
0925 22:55:16.393030 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:16.393068 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:16.394194 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:16.394670 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:16.394676 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:16.394766 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:16.395381 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:55:16.395382 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:16.395423 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:55:16.395492 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:55:16.395720 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:55:16.395725 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:55:16.395728 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 22:55:16.395848 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 22:55:16.396062 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":118,"threads_started":1}
I20260925 22:55:16.398012 31021 heartbeater.cc:347] Connected to a master server at 127.20.114.254:40037
I20260925 22:55:16.398243 31021 heartbeater.cc:464] Registering TS with master...
I20260925 22:55:16.398653 31021 heartbeater.cc:511] Master 127.20.114.254:40037 requested a full tablet report, sending...
I20260925 22:55:16.399665 30702 ts_manager.cc:194] Registered new tserver with Master: b622dc6f413340dbbac6ed4e3e9177b4 (127.20.114.194:35803)
I20260925 22:55:16.399981 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 30893
I20260925 22:55:16.400107 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:55:16.400439 20939 external_mini_cluster.cc:1428] Setting key 9534f8f888089b3a2bfae11c7688a245
I20260925 22:55:16.402277 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:40037
--builtin_ntp_servers=127.20.114.212:44245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 22:55:16.513096 31024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:55:16.513358 31024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:55:16.513432 31024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:55:16.517603 31024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:55:16.517663 31024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:55:16.517760 31024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:55:16.522523 31024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:44245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:40037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:55:16.523612 31024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:55:16.525104 31024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:55:16.534960 31029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:55:16.534973 31030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:55:16.534968 31032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:55:16.535039 31024 server_base.cc:1034] running on GCE node
I20260925 22:55:16.535593 31024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:55:16.536223 31024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:55:16.537420 31024 hybrid_clock.cc:648] HybridClock initialized: now 1790376916537409 us; error 28 us; skew 500 ppm
I20260925 22:55:16.539182 31024 webserver.cc:460] Webserver started at http://127.20.114.195:42309/ using document root <none> and password file <none>
I20260925 22:55:16.539775 31024 fs_manager.cc:359] Metadata directory not provided
I20260925 22:55:16.539875 31024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:55:16.540112 31024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:55:16.542201 31024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "5f9700bded9c4b06bc422fbce9570982"
format_stamp: "Formatted at 2026-09-25 22:55:16 on dist-test-slave-x1lh"
server_key: "0b33cde15aa37ebd3d619201d5fce6cf"
server_key_iv: "c4adf238f730d3c5f076f62b731581d6"
server_key_version: "encryptionkey@0"
I20260925 22:55:16.542773 31024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "5f9700bded9c4b06bc422fbce9570982"
format_stamp: "Formatted at 2026-09-25 22:55:16 on dist-test-slave-x1lh"
server_key: "0b33cde15aa37ebd3d619201d5fce6cf"
server_key_iv: "c4adf238f730d3c5f076f62b731581d6"
server_key_version: "encryptionkey@0"
I20260925 22:55:16.546700 31024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:55:16.549212 31038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:16.550390 31024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:55:16.550526 31024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "5f9700bded9c4b06bc422fbce9570982"
format_stamp: "Formatted at 2026-09-25 22:55:16 on dist-test-slave-x1lh"
server_key: "0b33cde15aa37ebd3d619201d5fce6cf"
server_key_iv: "c4adf238f730d3c5f076f62b731581d6"
server_key_version: "encryptionkey@0"
I20260925 22:55:16.550645 31024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:55:16.555162 31024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:55:16.555765 31024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:55:16.555949 31024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:55:16.556499 31024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:55:16.557448 31024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:55:16.557518 31024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:16.557585 31024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:55:16.557633 31024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:16.565984 31024 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:34201
I20260925 22:55:16.566002 31151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:34201 every 8 connection(s)
I20260925 22:55:16.566859 31024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:55:16.568190 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 31024
I20260925 22:55:16.568316 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:55:16.568581 20939 external_mini_cluster.cc:1428] Setting key 2119e7cb70895497174bb82bffd6cce5
I20260925 22:55:16.572955 30741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:16.568793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:41887 (local address 127.20.114.254:40037)
0925 22:55:16.568966 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:16.568970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:16.570171 (+ 1201us) server_negotiation.cc:407] Connection header received
0925 22:55:16.571465 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:16.571468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:16.571504 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:16.571568 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:16.572504 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:55:16.572525 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:55:16.572530 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:55:16.572544 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:55:16.572573 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:55:16.572582 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:55:16.572649 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 22:55:16.572742 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 22:55:16.572788 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:55:16.573792 31154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:16.569261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:40037 (local address 127.20.114.195:41887)
0925 22:55:16.570032 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:16.570082 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:16.571273 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:16.571694 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:16.571701 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:16.571785 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:16.572270 (+ 485us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:55:16.572272 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:16.572309 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:55:16.572380 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:55:16.572637 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:55:16.572642 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:55:16.572645 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 22:55:16.572759 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 22:55:16.573038 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":137,"threads_started":1}
I20260925 22:55:16.574870 31152 heartbeater.cc:347] Connected to a master server at 127.20.114.254:40037
I20260925 22:55:16.575090 31152 heartbeater.cc:464] Registering TS with master...
I20260925 22:55:16.575471 31152 heartbeater.cc:511] Master 127.20.114.254:40037 requested a full tablet report, sending...
I20260925 22:55:16.576396 30702 ts_manager.cc:194] Registered new tserver with Master: 5f9700bded9c4b06bc422fbce9570982 (127.20.114.195:34201)
I20260925 22:55:16.582118 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 22:55:16.585024 30741 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:55:16.583988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47290 (local address 127.20.114.254:40037)
0925 22:55:16.584166 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:16.584171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:16.584316 (+ 145us) server_negotiation.cc:407] Connection header received
0925 22:55:16.584492 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:16.584496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:16.584537 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:16.584621 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:16.584853 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47290: BlockingRecv error: recv got EOF from 127.0.0.1:47290 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 22:55:16.585189 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 30762
I20260925 22:55:16.591038 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 30893
I20260925 22:55:16.596199 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 31024
I20260925 22:55:16.601217 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 30671
2026-09-25T22:55:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (807 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:55:16 dist-test-slave-x1lh krb5kdc[31162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:55:16 dist-test-slave-x1lh krb5kdc[31162](info): set up 2 sockets
Sep 25 22:55:16 dist-test-slave-x1lh krb5kdc[31162](info): commencing operation
krb5kdc: starting...
W20260925 22:55:17.673780 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:55:17 dist-test-slave-x1lh krb5kdc[31162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:55:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:55:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:55:17.795586 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:44377
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:44377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:55:17.904191 31175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:55:17.904512 31175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:55:17.904588 31175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:55:17.908584 31175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:55:17.908679 31175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:55:17.908717 31175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:55:17.908769 31175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:55:17.908807 31175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:55:17.914027 31175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:44377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:44377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:55:17.915069 31175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:55:17.916105 31175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:55:17.924655 31183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:55:17.924659 31181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:55:17.924669 31180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:55:17.924906 31175 server_base.cc:1034] running on GCE node
I20260925 22:55:17.925583 31175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:55:17.926657 31175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:55:17.927860 31175 hybrid_clock.cc:648] HybridClock initialized: now 1790376917927852 us; error 28 us; skew 500 ppm
Sep 25 22:55:17 dist-test-slave-x1lh krb5kdc[31162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376917, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:55:17.934912 31175 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:55:17.935810 31175 webserver.cc:460] Webserver started at http://127.20.114.254:44597/ using document root <none> and password file <none>
I20260925 22:55:17.936373 31175 fs_manager.cc:359] Metadata directory not provided
I20260925 22:55:17.936465 31175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:55:17.936694 31175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:55:17.938768 31175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "2abb68ac673a4f71a306f7c00d3c0350"
format_stamp: "Formatted at 2026-09-25 22:55:17 on dist-test-slave-x1lh"
server_key: "5d4934b0eb4ea644835dd38e82cb0096"
server_key_iv: "97ececb93807dac6d3d40085b87f9d8c"
server_key_version: "encryptionkey@0"
I20260925 22:55:17.939352 31175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "2abb68ac673a4f71a306f7c00d3c0350"
format_stamp: "Formatted at 2026-09-25 22:55:17 on dist-test-slave-x1lh"
server_key: "5d4934b0eb4ea644835dd38e82cb0096"
server_key_iv: "97ececb93807dac6d3d40085b87f9d8c"
server_key_version: "encryptionkey@0"
I20260925 22:55:17.942955 31175 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 22:55:17.945246 31190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:17.946403 31175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:55:17.946542 31175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "2abb68ac673a4f71a306f7c00d3c0350"
format_stamp: "Formatted at 2026-09-25 22:55:17 on dist-test-slave-x1lh"
server_key: "5d4934b0eb4ea644835dd38e82cb0096"
server_key_iv: "97ececb93807dac6d3d40085b87f9d8c"
server_key_version: "encryptionkey@0"
I20260925 22:55:17.946648 31175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:55:17.967495 31175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:55:17.970402 31175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:55:17.970606 31175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:55:17.977632 31175 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:44377
I20260925 22:55:17.977936 31242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:44377 every 8 connection(s)
I20260925 22:55:17.978675 31175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:55:17.981374 31243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:55:17.981544 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 31175
I20260925 22:55:17.981653 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:55:17.981961 20939 external_mini_cluster.cc:1428] Setting key 77631e9ac1648c6ea977f9a4a8e12abc
I20260925 22:55:17.987625 31243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350: Bootstrap starting.
I20260925 22:55:17.989960 31243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350: Neither blocks nor log segments found. Creating new log.
Sep 25 22:55:17 dist-test-slave-x1lh krb5kdc[31162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:55:17.990845 31243 log.cc:824] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350: Log is configured to *not* fsync() on all Append() calls
I20260925 22:55:17.992913 31243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350: No bootstrap required, opened a new log
I20260925 22:55:17.995868 31246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:17.983338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47156 (local address 127.20.114.254:44377)
0925 22:55:17.983756 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:17.983771 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:17.983804 (+ 33us) server_negotiation.cc:407] Connection header received
0925 22:55:17.984585 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:17.984602 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:17.984998 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:17.985332 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:17.986144 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:17.987099 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:17.988024 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:17.988486 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:17.988660 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:17.991408 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:55:17.991413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:55:17.991425 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:55:17.991451 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:55:17.993159 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:55:17.993610 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:55:17.993616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:55:17.993622 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:55:17.993737 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:55:17.994013 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:55:17.994017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:55:17.994019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:55:17.994404 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:55:17.994615 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 22:55:17.994861 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 22:55:17.995147 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":139,"threads_started":1}
I20260925 22:55:17.996464 31243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abb68ac673a4f71a306f7c00d3c0350" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44377 } }
I20260925 22:55:17.996677 31243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:55:17.996711 31243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2abb68ac673a4f71a306f7c00d3c0350, State: Initialized, Role: FOLLOWER
I20260925 22:55:17.997264 31243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abb68ac673a4f71a306f7c00d3c0350" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44377 } }
I20260925 22:55:17.997419 31243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:55:17.997489 31243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:55:17.997581 31243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:55:17.998643 31243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abb68ac673a4f71a306f7c00d3c0350" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44377 } }
I20260925 22:55:17.998929 31243 leader_election.cc:302] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2abb68ac673a4f71a306f7c00d3c0350; no voters:
I20260925 22:55:17.999173 31243 leader_election.cc:288] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:55:17.999301 31248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:55:17.999539 31248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350 [term 1 LEADER]: Becoming Leader. State: Replica: 2abb68ac673a4f71a306f7c00d3c0350, State: Running, Role: LEADER
I20260925 22:55:17.999912 31248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abb68ac673a4f71a306f7c00d3c0350" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44377 } }
I20260925 22:55:18.000406 31243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:55:18.001515 31250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2abb68ac673a4f71a306f7c00d3c0350. Latest consensus state: current_term: 1 leader_uuid: "2abb68ac673a4f71a306f7c00d3c0350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abb68ac673a4f71a306f7c00d3c0350" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44377 } } }
I20260925 22:55:18.001632 31250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350 [sys.catalog]: This master's current role is: LEADER
I20260925 22:55:18.002035 31249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2abb68ac673a4f71a306f7c00d3c0350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abb68ac673a4f71a306f7c00d3c0350" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44377 } } }
I20260925 22:55:18.002198 31249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350 [sys.catalog]: This master's current role is: LEADER
W20260925 22:55:18.004838 31263 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:55:18.004969 31263 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:55:18.005048 31257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:55:18.005671 31257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:55:18.010967 31257 catalog_manager.cc:1347] Generated new cluster ID: e97397f5747249c5a2a9abeee270c821
I20260925 22:55:18.011024 31257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:55:18.020040 31257 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:55:18.021214 31257 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:55:18.026813 31257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350: Generated new TSK 0
I20260925 22:55:18.027550 31257 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.114.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
I20260925 22:55:18.056025 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44377
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:55:18.168383 31269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:55:18.168672 31269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:55:18.168747 31269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:55:18.172812 31269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:55:18.172907 31269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:55:18.173017 31269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:55:18.178254 31269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:55:18.179286 31269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:55:18.180333 31269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:55:18.191273 31274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:55:18.191511 31275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:55:18.191540 31277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:55:18.191753 31269 server_base.cc:1034] running on GCE node
I20260925 22:55:18.192238 31269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:55:18.193042 31269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:55:18.194274 31269 hybrid_clock.cc:648] HybridClock initialized: now 1790376918194253 us; error 36 us; skew 500 ppm
Sep 25 22:55:18 dist-test-slave-x1lh krb5kdc[31162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376918, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:55:18.197839 31269 init.cc:377] Logged in from keytab as kudu/127.20.114.193@KRBTEST.COM (short username kudu)
I20260925 22:55:18.198814 31269 webserver.cc:460] Webserver started at http://127.20.114.193:40249/ using document root <none> and password file <none>
I20260925 22:55:18.199415 31269 fs_manager.cc:359] Metadata directory not provided
I20260925 22:55:18.199504 31269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:55:18.199738 31269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:55:18.202097 31269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "b0405be9187046eabd76f1f1ad6194cd"
format_stamp: "Formatted at 2026-09-25 22:55:18 on dist-test-slave-x1lh"
server_key: "69c8bf9f37f0851c2fca05dabf1c9b25"
server_key_iv: "99613df3651a5c250f7f1839e1278179"
server_key_version: "encryptionkey@0"
I20260925 22:55:18.202680 31269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "b0405be9187046eabd76f1f1ad6194cd"
format_stamp: "Formatted at 2026-09-25 22:55:18 on dist-test-slave-x1lh"
server_key: "69c8bf9f37f0851c2fca05dabf1c9b25"
server_key_iv: "99613df3651a5c250f7f1839e1278179"
server_key_version: "encryptionkey@0"
I20260925 22:55:18.206970 31269 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 22:55:18.209924 31284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:18.211439 31269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:55:18.211611 31269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "b0405be9187046eabd76f1f1ad6194cd"
format_stamp: "Formatted at 2026-09-25 22:55:18 on dist-test-slave-x1lh"
server_key: "69c8bf9f37f0851c2fca05dabf1c9b25"
server_key_iv: "99613df3651a5c250f7f1839e1278179"
server_key_version: "encryptionkey@0"
I20260925 22:55:18.211900 31269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:55:18.233793 31269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:55:18.236969 31269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:55:18.237176 31269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:55:18.237811 31269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:55:18.238663 31269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:55:18.238731 31269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:18.238808 31269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:55:18.238857 31269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:18.248318 31269 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:45871
I20260925 22:55:18.248409 31397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:45871 every 8 connection(s)
I20260925 22:55:18.249411 31269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:55:18.253088 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 31269
I20260925 22:55:18.253219 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:55:18.253496 20939 external_mini_cluster.cc:1428] Setting key 43e295b51ddaaf3605e02ff09536b10f
Sep 25 22:55:18 dist-test-slave-x1lh krb5kdc[31162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376918, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.114.194@KRBTEST.COM; defaulting to no policy
I20260925 22:55:18.266355 31246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:18.251543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:38403 (local address 127.20.114.254:44377)
0925 22:55:18.251722 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:18.251728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:18.252652 (+ 924us) server_negotiation.cc:407] Connection header received
0925 22:55:18.254262 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:18.254265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:18.254321 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:18.254413 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:18.255865 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.256744 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.258566 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.259050 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.259181 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:18.261900 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:55:18.261905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:55:18.261908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:55:18.261945 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:55:18.264029 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:55:18.264603 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:55:18.264606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:55:18.264607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:55:18.264695 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:55:18.265153 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:55:18.265157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:55:18.265159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:55:18.265408 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:55:18.265492 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 22:55:18.266098 (+ 606us) server_negotiation.cc:299] Negotiation successful
0925 22:55:18.266221 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.194@KRBTEST.COM" created.
I20260925 22:55:18.267407 31400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:18.251974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:44377 (local address 127.20.114.193:38403)
0925 22:55:18.252471 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:18.252534 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:18.253979 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:18.254624 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:18.254632 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:18.255143 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:18.255697 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:18.255713 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.256942 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.256947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:18.258416 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:18.258428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.259244 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.259248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:18.259405 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:18.260138 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:18.260156 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:18.261681 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:55:18.264162 (+ 2481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:55:18.264168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:55:18.264183 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:55:18.264480 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:55:18.264779 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:55:18.264781 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:55:18.264782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:55:18.265012 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:55:18.265510 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:55:18.265515 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:55:18.265781 (+ 266us) client_negotiation.cc:769] Sending connection context
0925 22:55:18.265971 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 22:55:18.266339 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":120,"threads_started":1}
I20260925 22:55:18.268817 31398 heartbeater.cc:347] Connected to a master server at 127.20.114.254:44377
I20260925 22:55:18.269106 31398 heartbeater.cc:464] Registering TS with master...
I20260925 22:55:18.269652 31398 heartbeater.cc:511] Master 127.20.114.254:44377 requested a full tablet report, sending...
I20260925 22:55:18.271363 31207 ts_manager.cc:194] Registered new tserver with Master: b0405be9187046eabd76f1f1ad6194cd (127.20.114.193:45871)
I20260925 22:55:18.273146 31207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:38403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
I20260925 22:55:18.289650 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44377
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:55:18.404633 31403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:55:18.404938 31403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:55:18.404983 31403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:55:18.409062 31403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:55:18.409133 31403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:55:18.409210 31403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:55:18.414544 31403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:55:18.415704 31403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:55:18.416795 31403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:55:18.426616 31411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:55:18.426630 31408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:55:18.426801 31403 server_base.cc:1034] running on GCE node
W20260925 22:55:18.426646 31409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:55:18.427210 31403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:55:18.427835 31403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:55:18.428942 31403 hybrid_clock.cc:648] HybridClock initialized: now 1790376918428920 us; error 29 us; skew 500 ppm
Sep 25 22:55:18 dist-test-slave-x1lh krb5kdc[31162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376918, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:55:18.431730 31403 init.cc:377] Logged in from keytab as kudu/127.20.114.194@KRBTEST.COM (short username kudu)
I20260925 22:55:18.432689 31403 webserver.cc:460] Webserver started at http://127.20.114.194:42885/ using document root <none> and password file <none>
I20260925 22:55:18.433323 31403 fs_manager.cc:359] Metadata directory not provided
I20260925 22:55:18.433419 31403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:55:18.433650 31403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:55:18.435700 31403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "ec3cb0fd780d42d48040b3ddca9f39dc"
format_stamp: "Formatted at 2026-09-25 22:55:18 on dist-test-slave-x1lh"
server_key: "462ede3213fe76d02db526c939f8f009"
server_key_iv: "81fada4a0e37e338e1a505aad98069ff"
server_key_version: "encryptionkey@0"
I20260925 22:55:18.436273 31403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "ec3cb0fd780d42d48040b3ddca9f39dc"
format_stamp: "Formatted at 2026-09-25 22:55:18 on dist-test-slave-x1lh"
server_key: "462ede3213fe76d02db526c939f8f009"
server_key_iv: "81fada4a0e37e338e1a505aad98069ff"
server_key_version: "encryptionkey@0"
I20260925 22:55:18.440037 31403 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:55:18.442581 31418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:18.443806 31403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:55:18.443951 31403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "ec3cb0fd780d42d48040b3ddca9f39dc"
format_stamp: "Formatted at 2026-09-25 22:55:18 on dist-test-slave-x1lh"
server_key: "462ede3213fe76d02db526c939f8f009"
server_key_iv: "81fada4a0e37e338e1a505aad98069ff"
server_key_version: "encryptionkey@0"
I20260925 22:55:18.444067 31403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:55:18.461100 31403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:55:18.464395 31403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:55:18.464612 31403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:55:18.465284 31403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:55:18.466125 31403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:55:18.466195 31403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:18.466261 31403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:55:18.466303 31403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:18.475106 31403 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:41601
I20260925 22:55:18.475128 31531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:41601 every 8 connection(s)
I20260925 22:55:18.476013 31403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:55:18.477236 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 31403
I20260925 22:55:18.477345 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:55:18.477667 20939 external_mini_cluster.cc:1428] Setting key 6c04f41839d45cfa079f0ce313d2da23
Sep 25 22:55:18 dist-test-slave-x1lh krb5kdc[31162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376918, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.114.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.195@KRBTEST.COM" created.
I20260925 22:55:18.491016 31246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:18.478127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:38869 (local address 127.20.114.254:44377)
0925 22:55:18.478252 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:18.478256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:18.479276 (+ 1020us) server_negotiation.cc:407] Connection header received
0925 22:55:18.480752 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:18.480755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:18.480809 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:18.480970 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:18.482175 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.482702 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.483860 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.484094 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.484209 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:18.487008 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:55:18.487012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:55:18.487015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:55:18.487044 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:55:18.488691 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:55:18.489272 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:55:18.489275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:55:18.489276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:55:18.489413 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:55:18.489807 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:55:18.489810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:55:18.489811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:55:18.490001 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:55:18.490088 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:55:18.490708 (+ 620us) server_negotiation.cc:299] Negotiation successful
0925 22:55:18.490890 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 22:55:18.492237 31534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:18.478650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:44377 (local address 127.20.114.194:38869)
0925 22:55:18.479131 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:18.479185 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:18.480532 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:18.481167 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:18.481174 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:18.481546 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:18.482025 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:18.482038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.482828 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.482832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:18.483720 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:18.483729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.484252 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.484256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:18.484459 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:18.485128 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:18.485145 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:18.486808 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:55:18.488806 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:55:18.488811 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:55:18.488824 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:55:18.489156 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:55:18.489515 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:55:18.489517 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:55:18.489519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:55:18.489672 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:55:18.490097 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:55:18.490102 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:55:18.490387 (+ 285us) client_negotiation.cc:769] Sending connection context
0925 22:55:18.490642 (+ 255us) client_negotiation.cc:240] Negotiation successful
0925 22:55:18.491013 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":134,"threads_started":1}
I20260925 22:55:18.493611 31532 heartbeater.cc:347] Connected to a master server at 127.20.114.254:44377
I20260925 22:55:18.493920 31532 heartbeater.cc:464] Registering TS with master...
I20260925 22:55:18.494443 31532 heartbeater.cc:511] Master 127.20.114.254:44377 requested a full tablet report, sending...
I20260925 22:55:18.495723 31207 ts_manager.cc:194] Registered new tserver with Master: ec3cb0fd780d42d48040b3ddca9f39dc (127.20.114.194:41601)
I20260925 22:55:18.496371 31207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:38869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
I20260925 22:55:18.509483 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44377
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:55:18.624360 31537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:55:18.624675 31537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:55:18.624748 31537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:55:18.629683 31537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:55:18.629777 31537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:55:18.629878 31537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:55:18.636089 31537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:55:18.637380 31537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:55:18.638484 31537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:55:18.648591 31543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:55:18.648591 31542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:55:18.648725 31537 server_base.cc:1034] running on GCE node
W20260925 22:55:18.648590 31545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:55:18.649410 31537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:55:18.650137 31537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:55:18.651304 31537 hybrid_clock.cc:648] HybridClock initialized: now 1790376918651284 us; error 35 us; skew 500 ppm
Sep 25 22:55:18 dist-test-slave-x1lh krb5kdc[31162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376918, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:55:18.654438 31537 init.cc:377] Logged in from keytab as kudu/127.20.114.195@KRBTEST.COM (short username kudu)
I20260925 22:55:18.655422 31537 webserver.cc:460] Webserver started at http://127.20.114.195:34629/ using document root <none> and password file <none>
I20260925 22:55:18.656076 31537 fs_manager.cc:359] Metadata directory not provided
I20260925 22:55:18.656143 31537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:55:18.656376 31537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:55:18.658638 31537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "fd71b6646186478396fd6aee25300dc8"
format_stamp: "Formatted at 2026-09-25 22:55:18 on dist-test-slave-x1lh"
server_key: "9fa9bb86b2e423361199b72e91efc16e"
server_key_iv: "ac4276b50274c7de4c88460c4acd9d83"
server_key_version: "encryptionkey@0"
I20260925 22:55:18.659232 31537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "fd71b6646186478396fd6aee25300dc8"
format_stamp: "Formatted at 2026-09-25 22:55:18 on dist-test-slave-x1lh"
server_key: "9fa9bb86b2e423361199b72e91efc16e"
server_key_iv: "ac4276b50274c7de4c88460c4acd9d83"
server_key_version: "encryptionkey@0"
I20260925 22:55:18.663264 31537 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:55:18.666002 31552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:18.667394 31537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:55:18.667555 31537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "fd71b6646186478396fd6aee25300dc8"
format_stamp: "Formatted at 2026-09-25 22:55:18 on dist-test-slave-x1lh"
server_key: "9fa9bb86b2e423361199b72e91efc16e"
server_key_iv: "ac4276b50274c7de4c88460c4acd9d83"
server_key_version: "encryptionkey@0"
I20260925 22:55:18.667682 31537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:55:18.690092 31537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:55:18.693756 31537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:55:18.694049 31537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:55:18.694689 31537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:55:18.695551 31537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:55:18.695623 31537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:18.695712 31537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:55:18.695761 31537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:55:18.705387 31537 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:46359
I20260925 22:55:18.705416 31665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:46359 every 8 connection(s)
I20260925 22:55:18.706331 31537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:55:18.706705 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 31537
I20260925 22:55:18.706841 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:55:18.707185 20939 external_mini_cluster.cc:1428] Setting key b58391ac98ce091c3bb39d04bbc5eb44
Sep 25 22:55:18 dist-test-slave-x1lh krb5kdc[31162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376918, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:55:18.720922 31246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:18.708547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:39807 (local address 127.20.114.254:44377)
0925 22:55:18.708714 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:18.708720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:18.709800 (+ 1080us) server_negotiation.cc:407] Connection header received
0925 22:55:18.711208 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:18.711211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:18.711255 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:18.711332 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:18.712497 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.713087 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.714276 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.714497 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.714580 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:18.717389 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:55:18.717392 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:55:18.717394 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:55:18.717419 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:55:18.718742 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:55:18.719229 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:55:18.719232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:55:18.719234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:55:18.719317 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:55:18.719614 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:55:18.719616 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:55:18.719617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:55:18.719914 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:55:18.720014 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:55:18.720549 (+ 535us) server_negotiation.cc:299] Negotiation successful
0925 22:55:18.720728 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:55:18.722033 31668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:18.709106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:44377 (local address 127.20.114.195:39807)
0925 22:55:18.709644 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:18.709697 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:18.711006 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:18.711472 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:18.711478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:18.711852 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:55:18.712346 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:18.712359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.713224 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.713226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:18.714129 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:18.714137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.714636 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.714639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:18.714788 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:18.715532 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:55:18.715548 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:55:18.717208 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:55:18.718854 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:55:18.718859 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:55:18.718871 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:55:18.719117 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:55:18.719402 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:55:18.719404 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:55:18.719406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:55:18.719533 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:55:18.720027 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:55:18.720033 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:55:18.720267 (+ 234us) client_negotiation.cc:769] Sending connection context
0925 22:55:18.720501 (+ 234us) client_negotiation.cc:240] Negotiation successful
0925 22:55:18.720911 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":135,"threads_started":1}
I20260925 22:55:18.723141 31666 heartbeater.cc:347] Connected to a master server at 127.20.114.254:44377
I20260925 22:55:18.723433 31666 heartbeater.cc:464] Registering TS with master...
I20260925 22:55:18.723945 31666 heartbeater.cc:511] Master 127.20.114.254:44377 requested a full tablet report, sending...
I20260925 22:55:18.725009 31207 ts_manager.cc:194] Registered new tserver with Master: fd71b6646186478396fd6aee25300dc8 (127.20.114.195:46359)
I20260925 22:55:18.725538 31207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:39807
I20260925 22:55:18.733943 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:55:18.743834 31246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:18.735925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47160 (local address 127.20.114.254:44377)
0925 22:55:18.736075 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:18.736080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:18.736327 (+ 247us) server_negotiation.cc:407] Connection header received
0925 22:55:18.736415 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:18.736417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:18.736459 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:18.736524 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:55:18.737331 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.737863 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.738724 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.738950 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.739029 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:18.740343 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:55:18.740346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:55:18.740348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:55:18.740372 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:55:18.741907 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:55:18.742359 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:55:18.742362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:55:18.742364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:55:18.742453 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:55:18.742770 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:55:18.742773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:55:18.742774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:55:18.743024 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:55:18.743139 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 22:55:18.743439 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 22:55:18.743604 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:55:18.747607 31207 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:55:18.751022 31207 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:55:18.766716 31680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:18.760822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52940 (local address 127.20.114.193:45871)
0925 22:55:18.761205 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:18.761222 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:18.761245 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:55:18.761324 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:18.761333 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:18.761519 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:18.761967 (+ 448us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:55:18.762123 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.763344 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.765351 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.765945 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.766089 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:18.766241 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 22:55:18.766423 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 22:55:18.766535 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":110,"threads_started":1}
I20260925 22:55:18.766706 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:18.760095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:45871 (local address 127.0.0.1:52940)
0925 22:55:18.760740 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:18.760753 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:18.760839 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:18.761675 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:18.761679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:18.761695 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:55:18.761824 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:18.761831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.763470 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.763476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:18.765200 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:18.765206 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.766078 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.766082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:18.766207 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:18.766244 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 22:55:18.766389 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 22:55:18.766540 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":264,"thread_start_us":124,"threads_started":1}
I20260925 22:55:18.766964 31679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:18.760096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60530 (local address 127.20.114.195:46359)
0925 22:55:18.760855 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:18.760897 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:18.760926 (+ 29us) server_negotiation.cc:407] Connection header received
0925 22:55:18.761013 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:18.761019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:18.761266 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:18.761846 (+ 580us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:55:18.762011 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.763352 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.765328 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.766103 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.766538 (+ 435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:18.766672 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 22:55:18.766725 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 22:55:18.766821 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":166,"threads_started":1}
I20260925 22:55:18.767037 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:18.759973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:46359 (local address 127.0.0.1:60530)
0925 22:55:18.760329 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:18.760379 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:18.760505 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:18.761460 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:18.761466 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:18.761490 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:55:18.761681 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:18.761688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.763545 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.763548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:18.765187 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:18.765199 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.766239 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.766243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:18.766336 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:18.766359 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:55:18.766410 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 22:55:18.766930 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":580,"thread_start_us":118,"threads_started":1}
I20260925 22:55:18.766714 31681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:18.760992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:41601 (local address 127.0.0.1:36802)
0925 22:55:18.761771 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:18.761782 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:18.761889 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:18.762287 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:18.762290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:18.762307 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:55:18.762422 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:18.762431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.763379 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.763382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:18.765003 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:18.765013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.766312 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.766315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:18.766409 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:18.766428 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 22:55:18.766489 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 22:55:18.766556 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":88,"thread_start_us":89,"threads_started":1}
I20260925 22:55:18.767508 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:18.761154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36802 (local address 127.20.114.194:41601)
0925 22:55:18.761595 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:18.761611 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:18.761874 (+ 263us) server_negotiation.cc:407] Connection header received
0925 22:55:18.761999 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:18.762004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:18.762146 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:18.762303 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:55:18.762516 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.763238 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.765124 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.765969 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.766096 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:18.766261 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 22:55:18.766772 (+ 511us) server_negotiation.cc:299] Negotiation successful
0925 22:55:18.766856 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":107,"threads_started":1}
I20260925 22:55:18.770036 31600 tablet_service.cc:1467] Processing CreateTablet for tablet 1806a376095f45b6bf625bc6b00c40f3 (DEFAULT_TABLE table=test-table [id=12fa7bfe841c4d3fbe61f891ccbdc7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:55:18.770196 31332 tablet_service.cc:1467] Processing CreateTablet for tablet 1806a376095f45b6bf625bc6b00c40f3 (DEFAULT_TABLE table=test-table [id=12fa7bfe841c4d3fbe61f891ccbdc7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:55:18.770196 31466 tablet_service.cc:1467] Processing CreateTablet for tablet 1806a376095f45b6bf625bc6b00c40f3 (DEFAULT_TABLE table=test-table [id=12fa7bfe841c4d3fbe61f891ccbdc7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:55:18.771183 31600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1806a376095f45b6bf625bc6b00c40f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:55:18.771211 31466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1806a376095f45b6bf625bc6b00c40f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:55:18.771211 31332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1806a376095f45b6bf625bc6b00c40f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:55:18.777810 31683 tablet_bootstrap.cc:492] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8: Bootstrap starting.
I20260925 22:55:18.779498 31685 tablet_bootstrap.cc:492] T 1806a376095f45b6bf625bc6b00c40f3 P ec3cb0fd780d42d48040b3ddca9f39dc: Bootstrap starting.
I20260925 22:55:18.780568 31683 tablet_bootstrap.cc:654] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8: Neither blocks nor log segments found. Creating new log.
I20260925 22:55:18.781225 31684 tablet_bootstrap.cc:492] T 1806a376095f45b6bf625bc6b00c40f3 P b0405be9187046eabd76f1f1ad6194cd: Bootstrap starting.
I20260925 22:55:18.781454 31683 log.cc:824] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8: Log is configured to *not* fsync() on all Append() calls
I20260925 22:55:18.782001 31685 tablet_bootstrap.cc:654] T 1806a376095f45b6bf625bc6b00c40f3 P ec3cb0fd780d42d48040b3ddca9f39dc: Neither blocks nor log segments found. Creating new log.
I20260925 22:55:18.782887 31685 log.cc:824] T 1806a376095f45b6bf625bc6b00c40f3 P ec3cb0fd780d42d48040b3ddca9f39dc: Log is configured to *not* fsync() on all Append() calls
I20260925 22:55:18.783501 31684 tablet_bootstrap.cc:654] T 1806a376095f45b6bf625bc6b00c40f3 P b0405be9187046eabd76f1f1ad6194cd: Neither blocks nor log segments found. Creating new log.
I20260925 22:55:18.783747 31683 tablet_bootstrap.cc:492] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8: No bootstrap required, opened a new log
I20260925 22:55:18.783958 31683 ts_tablet_manager.cc:1397] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 22:55:18.784341 31684 log.cc:824] T 1806a376095f45b6bf625bc6b00c40f3 P b0405be9187046eabd76f1f1ad6194cd: Log is configured to *not* fsync() on all Append() calls
I20260925 22:55:18.785121 31685 tablet_bootstrap.cc:492] T 1806a376095f45b6bf625bc6b00c40f3 P ec3cb0fd780d42d48040b3ddca9f39dc: No bootstrap required, opened a new log
I20260925 22:55:18.785324 31685 ts_tablet_manager.cc:1397] T 1806a376095f45b6bf625bc6b00c40f3 P ec3cb0fd780d42d48040b3ddca9f39dc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260925 22:55:18.786365 31684 tablet_bootstrap.cc:492] T 1806a376095f45b6bf625bc6b00c40f3 P b0405be9187046eabd76f1f1ad6194cd: No bootstrap required, opened a new log
I20260925 22:55:18.786552 31684 ts_tablet_manager.cc:1397] T 1806a376095f45b6bf625bc6b00c40f3 P b0405be9187046eabd76f1f1ad6194cd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:55:18.787496 31683 raft_consensus.cc:348] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71b6646186478396fd6aee25300dc8" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 46359 } } peers { permanent_uuid: "b0405be9187046eabd76f1f1ad6194cd" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45871 } } peers { permanent_uuid: "ec3cb0fd780d42d48040b3ddca9f39dc" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 41601 } }
I20260925 22:55:18.787810 31683 raft_consensus.cc:374] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:55:18.787887 31683 raft_consensus.cc:729] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd71b6646186478396fd6aee25300dc8, State: Initialized, Role: FOLLOWER
I20260925 22:55:18.788357 31683 consensus_queue.cc:260] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71b6646186478396fd6aee25300dc8" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 46359 } } peers { permanent_uuid: "b0405be9187046eabd76f1f1ad6194cd" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45871 } } peers { permanent_uuid: "ec3cb0fd780d42d48040b3ddca9f39dc" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 41601 } }
I20260925 22:55:18.789088 31666 heartbeater.cc:503] Master 127.20.114.254:44377 was elected leader, sending a full tablet report...
I20260925 22:55:18.789444 31683 ts_tablet_manager.cc:1428] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260925 22:55:18.789280 31685 raft_consensus.cc:348] T 1806a376095f45b6bf625bc6b00c40f3 P ec3cb0fd780d42d48040b3ddca9f39dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71b6646186478396fd6aee25300dc8" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 46359 } } peers { permanent_uuid: "b0405be9187046eabd76f1f1ad6194cd" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45871 } } peers { permanent_uuid: "ec3cb0fd780d42d48040b3ddca9f39dc" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 41601 } }
I20260925 22:55:18.789527 31685 raft_consensus.cc:374] T 1806a376095f45b6bf625bc6b00c40f3 P ec3cb0fd780d42d48040b3ddca9f39dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:55:18.789590 31685 raft_consensus.cc:729] T 1806a376095f45b6bf625bc6b00c40f3 P ec3cb0fd780d42d48040b3ddca9f39dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3cb0fd780d42d48040b3ddca9f39dc, State: Initialized, Role: FOLLOWER
I20260925 22:55:18.790032 31685 consensus_queue.cc:260] T 1806a376095f45b6bf625bc6b00c40f3 P ec3cb0fd780d42d48040b3ddca9f39dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71b6646186478396fd6aee25300dc8" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 46359 } } peers { permanent_uuid: "b0405be9187046eabd76f1f1ad6194cd" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45871 } } peers { permanent_uuid: "ec3cb0fd780d42d48040b3ddca9f39dc" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 41601 } }
I20260925 22:55:18.790992 31689 raft_consensus.cc:482] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:55:18.791074 31685 ts_tablet_manager.cc:1428] T 1806a376095f45b6bf625bc6b00c40f3 P ec3cb0fd780d42d48040b3ddca9f39dc: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260925 22:55:18.790942 31684 raft_consensus.cc:348] T 1806a376095f45b6bf625bc6b00c40f3 P b0405be9187046eabd76f1f1ad6194cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71b6646186478396fd6aee25300dc8" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 46359 } } peers { permanent_uuid: "b0405be9187046eabd76f1f1ad6194cd" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45871 } } peers { permanent_uuid: "ec3cb0fd780d42d48040b3ddca9f39dc" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 41601 } }
I20260925 22:55:18.791239 31684 raft_consensus.cc:374] T 1806a376095f45b6bf625bc6b00c40f3 P b0405be9187046eabd76f1f1ad6194cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:55:18.791188 31689 raft_consensus.cc:504] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71b6646186478396fd6aee25300dc8" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 46359 } } peers { permanent_uuid: "b0405be9187046eabd76f1f1ad6194cd" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45871 } } peers { permanent_uuid: "ec3cb0fd780d42d48040b3ddca9f39dc" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 41601 } }
I20260925 22:55:18.791324 31684 raft_consensus.cc:729] T 1806a376095f45b6bf625bc6b00c40f3 P b0405be9187046eabd76f1f1ad6194cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0405be9187046eabd76f1f1ad6194cd, State: Initialized, Role: FOLLOWER
I20260925 22:55:18.791555 31532 heartbeater.cc:503] Master 127.20.114.254:44377 was elected leader, sending a full tablet report...
I20260925 22:55:18.791793 31684 consensus_queue.cc:260] T 1806a376095f45b6bf625bc6b00c40f3 P b0405be9187046eabd76f1f1ad6194cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71b6646186478396fd6aee25300dc8" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 46359 } } peers { permanent_uuid: "b0405be9187046eabd76f1f1ad6194cd" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45871 } } peers { permanent_uuid: "ec3cb0fd780d42d48040b3ddca9f39dc" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 41601 } }
I20260925 22:55:18.792740 31684 ts_tablet_manager.cc:1428] T 1806a376095f45b6bf625bc6b00c40f3 P b0405be9187046eabd76f1f1ad6194cd: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:55:18.793154 31398 heartbeater.cc:503] Master 127.20.114.254:44377 was elected leader, sending a full tablet report...
I20260925 22:55:18.795085 31689 leader_election.cc:288] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0405be9187046eabd76f1f1ad6194cd (127.20.114.193:45871), ec3cb0fd780d42d48040b3ddca9f39dc (127.20.114.194:41601)
I20260925 22:55:18.798172 31668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:18.792555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:45871 (local address 127.20.114.195:49271)
0925 22:55:18.792829 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:18.792845 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:18.792944 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:18.794226 (+ 1282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:18.794230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:18.794248 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:55:18.794367 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:18.794374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.795347 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.795351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:18.796801 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:18.796811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.797753 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.797757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:18.797867 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:18.797909 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 22:55:18.797972 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 22:55:18.798044 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260925 22:55:18.798182 31680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:18.793253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:49271 (local address 127.20.114.193:45871)
0925 22:55:18.793995 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:18.794000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:18.794014 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:55:18.794059 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:18.794063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:18.794106 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:18.794201 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:55:18.794505 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.795237 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.796971 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.797585 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.797699 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:18.797761 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 22:55:18.798002 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 22:55:18.798069 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661}
I20260925 22:55:18.798751 31352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1806a376095f45b6bf625bc6b00c40f3" candidate_uuid: "fd71b6646186478396fd6aee25300dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0405be9187046eabd76f1f1ad6194cd" is_pre_election: true
I20260925 22:55:18.799052 31352 raft_consensus.cc:2454] T 1806a376095f45b6bf625bc6b00c40f3 P b0405be9187046eabd76f1f1ad6194cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd71b6646186478396fd6aee25300dc8 in term 0.
I20260925 22:55:18.799116 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:18.794929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:37785 (local address 127.20.114.194:41601)
0925 22:55:18.795059 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:18.795062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:18.795456 (+ 394us) server_negotiation.cc:407] Connection header received
0925 22:55:18.795543 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:18.795546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:18.795589 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:18.795681 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:55:18.795943 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.796788 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.797907 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.798520 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.798634 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:18.798681 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 22:55:18.798946 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 22:55:18.799010 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:55:18.799144 31693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:18.794619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:41601 (local address 127.20.114.195:37785)
0925 22:55:18.795356 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:18.795370 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:55:18.795463 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:55:18.795687 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:55:18.795690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:55:18.795705 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:55:18.795804 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:18.795811 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.796958 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.796961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:18.797783 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:55:18.797791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.798747 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.798750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:55:18.798821 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:18.798837 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 22:55:18.798932 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 22:55:18.799000 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":101,"threads_started":1}
I20260925 22:55:18.799583 31553 leader_election.cc:302] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0405be9187046eabd76f1f1ad6194cd, fd71b6646186478396fd6aee25300dc8; no voters:
I20260925 22:55:18.799870 31689 raft_consensus.cc:2781] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:55:18.799831 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1806a376095f45b6bf625bc6b00c40f3" candidate_uuid: "fd71b6646186478396fd6aee25300dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec3cb0fd780d42d48040b3ddca9f39dc" is_pre_election: true
I20260925 22:55:18.799966 31689 raft_consensus.cc:482] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:55:18.800026 31689 raft_consensus.cc:3037] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:55:18.800146 31486 raft_consensus.cc:2454] T 1806a376095f45b6bf625bc6b00c40f3 P ec3cb0fd780d42d48040b3ddca9f39dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd71b6646186478396fd6aee25300dc8 in term 0.
I20260925 22:55:18.801359 31689 raft_consensus.cc:504] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71b6646186478396fd6aee25300dc8" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 46359 } } peers { permanent_uuid: "b0405be9187046eabd76f1f1ad6194cd" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45871 } } peers { permanent_uuid: "ec3cb0fd780d42d48040b3ddca9f39dc" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 41601 } }
I20260925 22:55:18.801813 31689 leader_election.cc:288] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8 [CANDIDATE]: Term 1 election: Requested vote from peers b0405be9187046eabd76f1f1ad6194cd (127.20.114.193:45871), ec3cb0fd780d42d48040b3ddca9f39dc (127.20.114.194:41601)
I20260925 22:55:18.802175 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1806a376095f45b6bf625bc6b00c40f3" candidate_uuid: "fd71b6646186478396fd6aee25300dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec3cb0fd780d42d48040b3ddca9f39dc"
I20260925 22:55:18.802318 31486 raft_consensus.cc:3037] T 1806a376095f45b6bf625bc6b00c40f3 P ec3cb0fd780d42d48040b3ddca9f39dc [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:55:18.803409 31486 raft_consensus.cc:2454] T 1806a376095f45b6bf625bc6b00c40f3 P ec3cb0fd780d42d48040b3ddca9f39dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd71b6646186478396fd6aee25300dc8 in term 1.
I20260925 22:55:18.803709 31352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1806a376095f45b6bf625bc6b00c40f3" candidate_uuid: "fd71b6646186478396fd6aee25300dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0405be9187046eabd76f1f1ad6194cd"
I20260925 22:55:18.803825 31556 leader_election.cc:302] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec3cb0fd780d42d48040b3ddca9f39dc, fd71b6646186478396fd6aee25300dc8; no voters:
I20260925 22:55:18.803826 31352 raft_consensus.cc:3037] T 1806a376095f45b6bf625bc6b00c40f3 P b0405be9187046eabd76f1f1ad6194cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:55:18.803996 31689 raft_consensus.cc:2781] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:55:18.804114 31689 raft_consensus.cc:686] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8 [term 1 LEADER]: Becoming Leader. State: Replica: fd71b6646186478396fd6aee25300dc8, State: Running, Role: LEADER
I20260925 22:55:18.804436 31689 consensus_queue.cc:237] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71b6646186478396fd6aee25300dc8" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 46359 } } peers { permanent_uuid: "b0405be9187046eabd76f1f1ad6194cd" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45871 } } peers { permanent_uuid: "ec3cb0fd780d42d48040b3ddca9f39dc" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 41601 } }
I20260925 22:55:18.805063 31352 raft_consensus.cc:2454] T 1806a376095f45b6bf625bc6b00c40f3 P b0405be9187046eabd76f1f1ad6194cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd71b6646186478396fd6aee25300dc8 in term 1.
I20260925 22:55:18.807659 31206 catalog_manager.cc:5495] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd71b6646186478396fd6aee25300dc8 (127.20.114.195). New cstate: current_term: 1 leader_uuid: "fd71b6646186478396fd6aee25300dc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71b6646186478396fd6aee25300dc8" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 46359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0405be9187046eabd76f1f1ad6194cd" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec3cb0fd780d42d48040b3ddca9f39dc" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 41601 } health_report { overall_health: UNKNOWN } } }
I20260925 22:55:18.821597 31679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:18.817884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60544 (local address 127.20.114.195:46359)
0925 22:55:18.818021 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:55:18.818026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:55:18.818041 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:55:18.818155 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:55:18.818157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:55:18.818212 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:55:18.818328 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:55:18.818589 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.819120 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.820006 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:55:18.820226 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:55:18.820331 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:55:18.820665 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:55:18.821196 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:55:18.821328 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 22:55:18.821391 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 22:55:18.821451 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:55:18.827490 31352 raft_consensus.cc:1261] T 1806a376095f45b6bf625bc6b00c40f3 P b0405be9187046eabd76f1f1ad6194cd [term 1 FOLLOWER]: Refusing update from remote peer fd71b6646186478396fd6aee25300dc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:55:18.827488 31486 raft_consensus.cc:1261] T 1806a376095f45b6bf625bc6b00c40f3 P ec3cb0fd780d42d48040b3ddca9f39dc [term 1 FOLLOWER]: Refusing update from remote peer fd71b6646186478396fd6aee25300dc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:55:18.828249 31691 consensus_queue.cc:1035] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0405be9187046eabd76f1f1ad6194cd" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 45871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:55:18.828464 31689 consensus_queue.cc:1035] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec3cb0fd780d42d48040b3ddca9f39dc" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:55:18.834427 31696 mvcc.cc:205] Tried to move back new op lower bound from 7333383859510312960 to 7333383859426062336. Current Snapshot: MvccSnapshot[applied={T|T < 7333383859510312960}]
I20260925 22:55:18.836469 31697 mvcc.cc:205] Tried to move back new op lower bound from 7333383859510312960 to 7333383859426062336. Current Snapshot: MvccSnapshot[applied={T|T < 7333383859510312960}]
I20260925 22:55:18.844692 31207 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:55:18.844923 31207 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:55:18.847430 31207 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2abb68ac673a4f71a306f7c00d3c0350: Sending DeleteTablet for 3 replicas of tablet 1806a376095f45b6bf625bc6b00c40f3
I20260925 22:55:18.848281 31600 tablet_service.cc:1514] Processing DeleteTablet for tablet 1806a376095f45b6bf625bc6b00c40f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:55:18 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:60530
I20260925 22:55:18.848295 31332 tablet_service.cc:1514] Processing DeleteTablet for tablet 1806a376095f45b6bf625bc6b00c40f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:55:18 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:52940
I20260925 22:55:18.848780 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 31269
I20260925 22:55:18.849464 31466 tablet_service.cc:1514] Processing DeleteTablet for tablet 1806a376095f45b6bf625bc6b00c40f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:55:18 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:36802
I20260925 22:55:18.849982 31706 tablet_replica.cc:330] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8: stopping tablet replica
I20260925 22:55:18.850397 31706 raft_consensus.cc:2229] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:55:18.850641 31708 tablet_replica.cc:330] T 1806a376095f45b6bf625bc6b00c40f3 P ec3cb0fd780d42d48040b3ddca9f39dc: stopping tablet replica
I20260925 22:55:18.850769 31706 raft_consensus.cc:2258] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:55:18.850962 31708 raft_consensus.cc:2229] T 1806a376095f45b6bf625bc6b00c40f3 P ec3cb0fd780d42d48040b3ddca9f39dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:55:18.851426 31708 raft_consensus.cc:2258] T 1806a376095f45b6bf625bc6b00c40f3 P ec3cb0fd780d42d48040b3ddca9f39dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:55:18.852334 31706 ts_tablet_manager.cc:1905] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:55:18.852981 31708 ts_tablet_manager.cc:1905] T 1806a376095f45b6bf625bc6b00c40f3 P ec3cb0fd780d42d48040b3ddca9f39dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:55:18.855217 31706 ts_tablet_manager.cc:1918] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:55:18.855436 31706 log.cc:1196] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/1806a376095f45b6bf625bc6b00c40f3
I20260925 22:55:18.855886 31706 ts_tablet_manager.cc:1939] T 1806a376095f45b6bf625bc6b00c40f3 P fd71b6646186478396fd6aee25300dc8: Deleting consensus metadata
I20260925 22:55:18.856432 31708 ts_tablet_manager.cc:1918] T 1806a376095f45b6bf625bc6b00c40f3 P ec3cb0fd780d42d48040b3ddca9f39dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:55:18.856618 31708 log.cc:1196] T 1806a376095f45b6bf625bc6b00c40f3 P ec3cb0fd780d42d48040b3ddca9f39dc: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/1806a376095f45b6bf625bc6b00c40f3
I20260925 22:55:18.857038 31708 ts_tablet_manager.cc:1939] T 1806a376095f45b6bf625bc6b00c40f3 P ec3cb0fd780d42d48040b3ddca9f39dc: Deleting consensus metadata
I20260925 22:55:18.857332 31194 catalog_manager.cc:4841] TS fd71b6646186478396fd6aee25300dc8 (127.20.114.195:46359): tablet 1806a376095f45b6bf625bc6b00c40f3 (table test-table [id=12fa7bfe841c4d3fbe61f891ccbdc7b6]) successfully deleted
W20260925 22:55:18.858249 31191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.114.193:45871 (error 108)
I20260925 22:55:18.858346 31194 catalog_manager.cc:4841] TS ec3cb0fd780d42d48040b3ddca9f39dc (127.20.114.194:41601): tablet 1806a376095f45b6bf625bc6b00c40f3 (table test-table [id=12fa7bfe841c4d3fbe61f891ccbdc7b6]) successfully deleted
I20260925 22:55:18.859182 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 31403
I20260925 22:55:18.859395 31681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:55:18.859072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:45871 (local address 127.0.0.1:52954)
0925 22:55:18.859186 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 22:55:18.859293 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.193:45871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 22:55:18.860832 31191 catalog_manager.cc:4584] TS b0405be9187046eabd76f1f1ad6194cd (127.20.114.193:45871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1806a376095f45b6bf625bc6b00c40f3: Network error: Client connection negotiation failed: client connection to 127.20.114.193:45871: connect: Connection refused (error 111)
I20260925 22:55:18.865401 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 31537
I20260925 22:55:18.871037 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 31175
2026-09-25T22:55:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2271 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:55:18 dist-test-slave-x1lh krb5kdc[31712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:55:18 dist-test-slave-x1lh krb5kdc[31712](info): set up 2 sockets
Sep 25 22:55:18 dist-test-slave-x1lh krb5kdc[31712](info): commencing operation
krb5kdc: starting...
W20260925 22:55:19.958563 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:55:20 dist-test-slave-x1lh krb5kdc[31712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:55:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:55:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.114.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.114.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.114.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/rangeradmin_127.20.114.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.114.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/rangeradmin_127.20.114.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.114.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.114.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.114.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/rangerlookup_127.20.114.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.114.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/rangerlookup_127.20.114.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.114.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.114.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.114.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/HTTP_127.20.114.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.114.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/HTTP_127.20.114.212@KRBTEST.COM.keytab.
I20260925 22:55:20.148182 20939 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskV8JYeA/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/postgres -l logfile start
2026-09-25 22:55:22.996 UTC [31742] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 22:55:22.997 UTC [31742] LOG: listening on IPv4 address "127.20.114.212", port 48407
2026-09-25 22:55:23.001 UTC [31742] LOG: listening on Unix socket "/tmp/.s.PGSQL.48407"
2026-09-25 22:55:23.024 UTC [31745] LOG: database system was shut down at 2026-09-25 22:55:21 UTC
2026-09-25 22:55:23.028 UTC [31742] LOG: database system is ready to accept connections
I20260925 22:55:23.997704 20939 mini_postgres.cc:96] Postgres bound to 48407
2026-09-25 22:55:24.004 UTC [31753] FATAL: database "slave" does not exist
127.20.114.212:48407 - accepting connections
I20260925 22:55:24.005053 20939 mini_ranger.cc:161] Starting Ranger...
I20260925 22:55:24.020592 20939 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 22:55:24.518455 20939 mini_ranger.cc:87] Created ranger Postgres database
I20260925 22:55:24.518544 20939 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ranger-admin
2026-09-25 22:55:24,795 [I] DB FLAVOR :POSTGRES
2026-09-25 22:55:24,796 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:55:24,796 [I] Checking connection..
2026-09-25 22:55:24,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:48407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 22:55:25,056 [I] Checking connection passed.
2026-09-25 22:55:25,056 [I] --------- Verifying version history table ---------
2026-09-25 22:55:25,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:48407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:55:25,308 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:55:25,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:48407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:55:25,574 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:55:25,574 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 22:55:25,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:48407/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 22:55:25.834 UTC [31817] WARNING: there is no transaction in progress
2026-09-25 22:55:25,840 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 22:55:25,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:48407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:55:26,104 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 22:55:26,105 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:55:26,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:48407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 22:55:26,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:48407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 22:55:26,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:48407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-x1lh.c.gcp-upstream.internal','N') ;"
2026-09-25 22:55:26,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:48407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:55:27,096 [I] Table x_portal_user does not exist in database ranger
2026-09-25 22:55:27,096 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:48407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:55:27,359 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 22:55:27,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:48407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:55:27,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:48407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:55:27,835 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 22:55:27,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:48407/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 22:55:28.791 UTC [31925] WARNING: there is no transaction in progress
2026-09-25 22:55:28.808 UTC [31925] WARNING: there is no transaction in progress
2026-09-25 22:55:28.827 UTC [31925] WARNING: there is no transaction in progress
2026-09-25 22:55:28.846 UTC [31925] WARNING: there is no transaction in progress
2026-09-25 22:55:28.865 UTC [31925] WARNING: there is no transaction in progress
2026-09-25 22:55:28.885 UTC [31925] WARNING: there is no transaction in progress
2026-09-25 22:55:28.943 UTC [31925] WARNING: there is no transaction in progress
2026-09-25 22:55:28.952 UTC [31925] WARNING: there is no transaction in progress
2026-09-25 22:55:28.968 UTC [31925] WARNING: there is no transaction in progress
2026-09-25 22:55:28.979 UTC [31925] WARNING: there is no transaction in progress
2026-09-25 22:55:28.991 UTC [31925] WARNING: there is no transaction in progress
2026-09-25 22:55:29.006 UTC [31925] WARNING: there is no transaction in progress
2026-09-25 22:55:29.016 UTC [31925] WARNING: there is no transaction in progress
2026-09-25 22:55:29.023 UTC [31925] WARNING: there is no transaction in progress
2026-09-25 22:55:29.031 UTC [31925] WARNING: there is no transaction in progress
2026-09-25 22:55:29.039 UTC [31925] WARNING: there is no transaction in progress
2026-09-25 22:55:29,713 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 22:55:29,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:48407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 22:55:29,967 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 22:55:29,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:48407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:55:30,219 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 22:55:30,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:48407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:55:30,471 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 22:55:30,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:48407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:55:30,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:48407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:55:30,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:48407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-x1lh.c.gcp-upstream.internal';"
2026-09-25 22:55:31,173 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 22:55:31,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:48407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 22:55:31,413 [I] DB_PATCHES have already been applied
I20260925 22:55:31.418643 20939 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ranger-admin:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskV8JYeA/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 22:55:31.418712 20939 mini_ranger.cc:193] Using host: 127.20.114.212
I20260925 22:55:31.422168 20939 mini_ranger.cc:227] Ranger admin URL: http://127.20.114.212:51815
Sep 25, 2026 10:55:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:55:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:55:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:55:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:55:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:55:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:55:32 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 22:55:32.534940 20939 mini_ranger.cc:160] Time spent starting Ranger: real 8.530s user 0.003s sys 0.007s
Sep 25, 2026 10:55:52 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:55:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 10:55:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:55:52 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:55:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:55:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:55:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:55:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:55:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:55:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:55:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:55:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:55:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:55:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:55:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:55:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:55:52 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:55:53 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 22:55:54.062036 20939 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.114.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.114.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.114.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/rangerkms_127.20.114.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.114.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/rangerkms_127.20.114.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.114.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.114.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.114.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/HTTP_127.20.114.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.114.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/HTTP_127.20.114.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 22:55:54 dist-test-slave-x1lh krb5kdc[31712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376954, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 22:55:54.158834 20939 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 22:55:54.169665 20939 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 22:55:55.312539 20939 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 22:55:55.312646 20939 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ranger-kms
I20260925 22:55:55.312757 20939 mini_ranger_kms.cc:227] Using postgres at 127.20.114.212:48407
2026-09-25 22:55:55,519 [I] DB FLAVOR :POSTGRES
2026-09-25 22:55:55,519 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:55:55,519 [I] Checking connection
2026-09-25 22:55:55,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:48407/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 22:55:55,767 [I] connection success
2026-09-25 22:55:55,768 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 22:55:55,768 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 22:55:55,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:48407/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 22:55:56,052 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 22:55:56,052 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:55:56,052 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 22:55:56,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:48407/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 22:55:56,363 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 22:55:56.905736 20939 mini_ranger_kms.cc:310] Created kms service
I20260925 22:55:57.114281 20939 mini_ranger_kms.cc:326] Created kudu user
I20260925 22:55:57.165133 20939 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 22:55:57 dist-test-slave-x1lh krb5kdc[31712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376954, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.114.212@KRBTEST.COM
I20260925 22:55:57.618518 20939 mini_ranger_kms.cc:384] Added ranger policy
I20260925 22:55:57.618692 20939 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ranger-kms:/tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 22:55:57.618716 20939 mini_ranger_kms.cc:240] Using host: 127.20.114.212
I20260925 22:55:57.622411 20939 mini_ranger_kms.cc:276] Ranger KMS PID: 32105
I20260925 22:55:57.622567 20939 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.114.212:48297
Sep 25, 2026 10:55:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:55:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:55:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:55:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:55:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:55:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 22:55:59.939242 20939 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.780s user 0.020s sys 0.170s
I20260925 22:55:59.939404 20939 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 22:55:59.939540 20939 mini_ranger_kms.cc:401] 127.20.114.212:48297/kms/v1/keys
Sep 25 22:56:00 dist-test-slave-x1lh krb5kdc[31712](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790376960, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.114.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 22:56:00 dist-test-slave-x1lh krb5kdc[31712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376960, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.114.212@KRBTEST.COM for HTTP/127.20.114.212@KRBTEST.COM
Sep 25 22:56:00 dist-test-slave-x1lh krb5kdc[31712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376960, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.114.212@KRBTEST.COM for HTTP/127.20.114.212@KRBTEST.COM
Sep 25 22:56:03 dist-test-slave-x1lh krb5kdc[31712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:56:03.590776 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:43839
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:43839
--ranger_config_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.114.212:48297/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:56:03.743723 32156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:56:03.744081 32156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:56:03.744158 32156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:56:03.748338 32156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:56:03.748459 32156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:56:03.748497 32156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:56:03.748533 32156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:56:03.748590 32156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:56:03.754489 32156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.114.212:48297/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:43839
--ranger_config_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:43839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:56:03.755759 32156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:56:03.756913 32156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:56:03.766831 32162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:56:03.766872 32156 server_base.cc:1034] running on GCE node
W20260925 22:56:03.766822 32161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:56:03.766999 32164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:56:03.767966 32156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:56:03.769459 32156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:56:03.770689 32156 hybrid_clock.cc:648] HybridClock initialized: now 1790376963770672 us; error 38 us; skew 500 ppm
Sep 25 22:56:03 dist-test-slave-x1lh krb5kdc[31712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376963, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:56:03.773993 32156 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:56:03.774947 32156 webserver.cc:460] Webserver started at http://127.20.114.254:37317/ using document root <none> and password file <none>
I20260925 22:56:03.775497 32156 fs_manager.cc:359] Metadata directory not provided
I20260925 22:56:03.775553 32156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:56:03.775728 32156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:56:03 dist-test-slave-x1lh krb5kdc[31712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376963, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for HTTP/127.20.114.212@KRBTEST.COM
I20260925 22:56:03.846079 32156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "c6429010135143fcbde03114b4d64b23"
format_stamp: "Formatted at 2026-09-25 22:56:03 on dist-test-slave-x1lh"
server_key: "41eeddfeb90970e7a86836537ade49f6"
server_key_iv: "7b924e87a819de2d1b1c1549350008f6"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:03.846801 32156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "c6429010135143fcbde03114b4d64b23"
format_stamp: "Formatted at 2026-09-25 22:56:03 on dist-test-slave-x1lh"
server_key: "41eeddfeb90970e7a86836537ade49f6"
server_key_iv: "7b924e87a819de2d1b1c1549350008f6"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:03.851372 32156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:56:03.871677 32171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:03.873718 32156 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:56:03.873910 32156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "c6429010135143fcbde03114b4d64b23"
format_stamp: "Formatted at 2026-09-25 22:56:03 on dist-test-slave-x1lh"
server_key: "41eeddfeb90970e7a86836537ade49f6"
server_key_iv: "7b924e87a819de2d1b1c1549350008f6"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:03.874089 32156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:56:03.889329 32156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:56:03.892716 32156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:56:03.892967 32156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:56:03.900988 32156 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:43839
I20260925 22:56:03.901013 32223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:43839 every 8 connection(s)
I20260925 22:56:03.902107 32156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:56:03.905440 32224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:56:03.911816 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 32156
I20260925 22:56:03.911970 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:56:03.912233 32224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23: Bootstrap starting.
I20260925 22:56:03.912276 20939 external_mini_cluster.cc:1428] Setting key 6bc4f7d493235acd82421c7950f463dc
I20260925 22:56:03.915071 32224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23: Neither blocks nor log segments found. Creating new log.
I20260925 22:56:03.915874 32224 log.cc:824] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23: Log is configured to *not* fsync() on all Append() calls
I20260925 22:56:03.918268 32224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23: No bootstrap required, opened a new log
Sep 25 22:56:03 dist-test-slave-x1lh krb5kdc[31712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:56:03.923099 32224 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6429010135143fcbde03114b4d64b23" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43839 } }
I20260925 22:56:03.923410 32224 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:56:03.923499 32224 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6429010135143fcbde03114b4d64b23, State: Initialized, Role: FOLLOWER
I20260925 22:56:03.924009 32224 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6429010135143fcbde03114b4d64b23" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43839 } }
I20260925 22:56:03.924190 32224 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:56:03.924269 32224 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:56:03.924404 32224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:56:03.925797 32224 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6429010135143fcbde03114b4d64b23" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43839 } }
I20260925 22:56:03.926222 32224 leader_election.cc:302] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6429010135143fcbde03114b4d64b23; no voters:
I20260925 22:56:03.926599 32224 leader_election.cc:288] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:56:03.926908 32229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:56:03.927269 32229 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23 [term 1 LEADER]: Becoming Leader. State: Replica: c6429010135143fcbde03114b4d64b23, State: Running, Role: LEADER
I20260925 22:56:03.927681 32229 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6429010135143fcbde03114b4d64b23" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43839 } }
I20260925 22:56:03.928083 32224 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:56:03.929617 32230 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6429010135143fcbde03114b4d64b23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6429010135143fcbde03114b4d64b23" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43839 } } }
I20260925 22:56:03.929742 32230 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23 [sys.catalog]: This master's current role is: LEADER
I20260925 22:56:03.930208 32231 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6429010135143fcbde03114b4d64b23. Latest consensus state: current_term: 1 leader_uuid: "c6429010135143fcbde03114b4d64b23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6429010135143fcbde03114b4d64b23" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43839 } } }
I20260925 22:56:03.930332 32227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:03.913970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33496 (local address 127.20.114.254:43839)
0925 22:56:03.914460 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0925 22:56:03.914482 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 22:56:03.914521 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:56:03.915771 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:56:03.915793 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:56:03.916211 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:56:03.916649 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:56:03.917677 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:03.918781 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:03.920043 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:03.920561 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:03.920696 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:03.923925 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:56:03.923932 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:56:03.923950 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:56:03.923996 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:56:03.926152 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:03.926746 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:03.926753 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:03.926759 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:03.926905 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:03.927252 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:03.927255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:03.927257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:03.927806 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:56:03.928057 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 22:56:03.928800 (+ 743us) server_negotiation.cc:299] Negotiation successful
0925 22:56:03.929178 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":156,"threads_started":1}
I20260925 22:56:03.930362 32231 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23 [sys.catalog]: This master's current role is: LEADER
I20260925 22:56:03.931233 32239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:56:03.930622 32224 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 22:56:03.937245 32239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:56:03.946452 32239 catalog_manager.cc:1347] Generated new cluster ID: caed56e2fee443249acfb6985d88339c
I20260925 22:56:03.946545 32239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:56:03.972889 32239 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:56:03.974459 32239 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:56:03.982771 32239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23: Generated new TSK 0
I20260925 22:56:03.983664 32239 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.114.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
I20260925 22:56:04.774346 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.114.212:48297/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:43839
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:56:04.909327 32269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:56:04.909734 32269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:56:04.909803 32269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:56:04.915620 32269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:56:04.915738 32269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:56:04.915835 32269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:56:04.921787 32269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.114.212:48297/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:43839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:56:04.922991 32269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:56:04.924170 32269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:56:04.935406 32277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:56:04.935416 32275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:56:04.936604 32274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:56:04.937036 32269 server_base.cc:1034] running on GCE node
I20260925 22:56:04.937623 32269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:56:04.938541 32269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:56:04.939873 32269 hybrid_clock.cc:648] HybridClock initialized: now 1790376964939840 us; error 49 us; skew 500 ppm
Sep 25 22:56:04 dist-test-slave-x1lh krb5kdc[31712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376964, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:56:04.944607 32269 init.cc:377] Logged in from keytab as kudu/127.20.114.193@KRBTEST.COM (short username kudu)
I20260925 22:56:04.945989 32269 webserver.cc:460] Webserver started at http://127.20.114.193:36203/ using document root <none> and password file <none>
I20260925 22:56:04.946854 32269 fs_manager.cc:359] Metadata directory not provided
I20260925 22:56:04.947026 32269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:56:04.947422 32269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:56:04 dist-test-slave-x1lh krb5kdc[31712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376964, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for HTTP/127.20.114.212@KRBTEST.COM
I20260925 22:56:04.967904 32269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "f141321b518941c79548115073ae6ac5"
format_stamp: "Formatted at 2026-09-25 22:56:04 on dist-test-slave-x1lh"
server_key: "0bae83893f06e3d9aa2f1a5186eaadd6"
server_key_iv: "a2d961e2625d3bf04611ede6e19e91e9"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:04.968957 32269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "f141321b518941c79548115073ae6ac5"
format_stamp: "Formatted at 2026-09-25 22:56:04 on dist-test-slave-x1lh"
server_key: "0bae83893f06e3d9aa2f1a5186eaadd6"
server_key_iv: "a2d961e2625d3bf04611ede6e19e91e9"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:04.975553 32269 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 22:56:04.998001 32286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:04.999585 32269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:56:04.999752 32269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "f141321b518941c79548115073ae6ac5"
format_stamp: "Formatted at 2026-09-25 22:56:04 on dist-test-slave-x1lh"
server_key: "0bae83893f06e3d9aa2f1a5186eaadd6"
server_key_iv: "a2d961e2625d3bf04611ede6e19e91e9"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:04.999909 32269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:56:05.027509 32269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:56:05.031783 32269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:56:05.032095 32269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:56:05.032991 32269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:56:05.034209 32269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:56:05.034333 32269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:05.034456 32269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:56:05.034540 32269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:05.048810 32269 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:43893
I20260925 22:56:05.049345 32401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:43893 every 8 connection(s)
I20260925 22:56:05.050648 32269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:56:05.057170 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 32269
I20260925 22:56:05.057327 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:56:05.057654 20939 external_mini_cluster.cc:1428] Setting key 2184a9a3152cc9f38005307bacc087fc
Sep 25 22:56:05 dist-test-slave-x1lh krb5kdc[31712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376964, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:56:05.074505 32404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:05.053110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:50101 (local address 127.20.114.254:43839)
0925 22:56:05.055028 (+ 1918us) server_negotiation.cc:206] Beginning negotiation
0925 22:56:05.055034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:56:05.056854 (+ 1820us) server_negotiation.cc:407] Connection header received
0925 22:56:05.056949 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:56:05.056953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:56:05.057015 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:56:05.057112 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:56:05.058990 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.059764 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.061646 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.062031 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.062112 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:05.068805 (+ 6693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:56:05.068811 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:56:05.068815 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:56:05.068856 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:56:05.071166 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:05.071997 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:05.072001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:05.072003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:05.072106 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:05.072565 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:05.072569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:05.072571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:05.072828 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:56:05.072962 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 22:56:05.074165 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0925 22:56:05.074310 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1823,"thread_start_us":83,"threads_started":1}
I20260925 22:56:05.076668 32405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:05.053817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43839 (local address 127.20.114.193:50101)
0925 22:56:05.055323 (+ 1506us) negotiation.cc:107] Waiting for socket to connect
0925 22:56:05.055388 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 22:56:05.056507 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:56:05.057609 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:56:05.057620 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:56:05.058078 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:56:05.058797 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:05.058817 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.059912 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.059918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:05.061486 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:05.061499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.062335 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.062339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:05.062558 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:05.063332 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:56:05.063356 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:56:05.065256 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:56:05.071373 (+ 6117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:56:05.071382 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:56:05.071405 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:56:05.071841 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:56:05.072228 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:56:05.072231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:56:05.072233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:56:05.072448 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:56:05.073030 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:56:05.073038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:56:05.073854 (+ 816us) client_negotiation.cc:769] Sending connection context
0925 22:56:05.075037 (+ 1183us) client_negotiation.cc:240] Negotiation successful
0925 22:56:05.075423 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1270,"thread_start_us":142,"threads_started":1}
WARNING: no policy specified for kudu/127.20.114.194@KRBTEST.COM; defaulting to no policy
I20260925 22:56:05.084627 32402 heartbeater.cc:347] Connected to a master server at 127.20.114.254:43839
I20260925 22:56:05.084968 32402 heartbeater.cc:464] Registering TS with master...
I20260925 22:56:05.085654 32402 heartbeater.cc:511] Master 127.20.114.254:43839 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.194@KRBTEST.COM" created.
I20260925 22:56:05.087663 32188 ts_manager.cc:194] Registered new tserver with Master: f141321b518941c79548115073ae6ac5 (127.20.114.193:43893)
I20260925 22:56:05.089479 32188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:50101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
I20260925 22:56:05.109747 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.114.212:48297/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:43839
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:56:05 dist-test-slave-x1lh krb5kdc[31712](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790376965, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 22:56:05.276710 32408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:56:05.277272 32408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:56:05.277617 32408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:56:05.284277 32408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:56:05.284380 32408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:56:05.284473 32408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:56:05.293087 32408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.114.212:48297/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:43839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:56:05.294621 32408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:56:05.296028 32408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:56:05.310496 32417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:56:05.310596 32415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:56:05.311110 32414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:56:05.311655 32408 server_base.cc:1034] running on GCE node
I20260925 22:56:05.312197 32408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:56:05.313108 32408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:56:05.314319 32408 hybrid_clock.cc:648] HybridClock initialized: now 1790376965314281 us; error 58 us; skew 500 ppm
Sep 25 22:56:05 dist-test-slave-x1lh krb5kdc[31712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376965, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:56:05.318347 32408 init.cc:377] Logged in from keytab as kudu/127.20.114.194@KRBTEST.COM (short username kudu)
I20260925 22:56:05.319659 32408 webserver.cc:460] Webserver started at http://127.20.114.194:45417/ using document root <none> and password file <none>
I20260925 22:56:05.320500 32408 fs_manager.cc:359] Metadata directory not provided
I20260925 22:56:05.320664 32408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:56:05.320982 32408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:56:05 dist-test-slave-x1lh krb5kdc[31712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376965, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for HTTP/127.20.114.212@KRBTEST.COM
I20260925 22:56:05.341326 32408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "c4215cb3a29246a1b20ef6185d77dc83"
format_stamp: "Formatted at 2026-09-25 22:56:05 on dist-test-slave-x1lh"
server_key: "66055d14b320bfec94a54aff38120545"
server_key_iv: "7102b1915dbfdd012e3526b3ec8f3b7d"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:05.342267 32408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "c4215cb3a29246a1b20ef6185d77dc83"
format_stamp: "Formatted at 2026-09-25 22:56:05 on dist-test-slave-x1lh"
server_key: "66055d14b320bfec94a54aff38120545"
server_key_iv: "7102b1915dbfdd012e3526b3ec8f3b7d"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:05.348554 32408 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 22:56:05.372229 32424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:05.374171 32408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:56:05.374338 32408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "c4215cb3a29246a1b20ef6185d77dc83"
format_stamp: "Formatted at 2026-09-25 22:56:05 on dist-test-slave-x1lh"
server_key: "66055d14b320bfec94a54aff38120545"
server_key_iv: "7102b1915dbfdd012e3526b3ec8f3b7d"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:05.374507 32408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:56:05.405998 32408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:56:05.417650 32408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:56:05.418143 32408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:56:05.419128 32408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:56:05.420449 32408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:56:05.420568 32408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:05.420665 32408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:56:05.420720 32408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:05.433375 32408 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:34937
I20260925 22:56:05.434789 32408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:56:05.441946 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 32408
I20260925 22:56:05.442101 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:56:05.442454 20939 external_mini_cluster.cc:1428] Setting key 4c2f773e990a95c6be8f60d512382f6f
I20260925 22:56:05.447170 32537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:34937 every 8 connection(s)
WARNING: no policy specified for kudu/127.20.114.195@KRBTEST.COM; defaulting to no policy
Sep 25 22:56:05 dist-test-slave-x1lh krb5kdc[31712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376965, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.195@KRBTEST.COM" created.
I20260925 22:56:05.477967 32404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:05.453691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:55869 (local address 127.20.114.254:43839)
0925 22:56:05.453868 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 22:56:05.453874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:56:05.454692 (+ 818us) server_negotiation.cc:407] Connection header received
0925 22:56:05.456128 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:56:05.456132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:56:05.456196 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:56:05.456254 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:56:05.458486 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.459437 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.461485 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.461918 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.462015 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:05.472586 (+ 10571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:56:05.472592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:56:05.472595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:56:05.472641 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:56:05.474763 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:05.475631 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:05.475636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:05.475638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:05.475740 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:05.476210 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:05.476214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:05.476216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:05.476466 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:56:05.476569 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:56:05.477588 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0925 22:56:05.477746 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:56:05.480479 32541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:05.452319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43839 (local address 127.20.114.194:55869)
0925 22:56:05.454496 (+ 2177us) negotiation.cc:107] Waiting for socket to connect
0925 22:56:05.454563 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 22:56:05.455812 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:56:05.456743 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:56:05.456754 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:56:05.457509 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:56:05.458278 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:05.458300 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.459628 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.459632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:05.461310 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:05.461323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.462257 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.462261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:05.462489 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:05.463280 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:56:05.463304 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:56:05.469989 (+ 6685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:56:05.474987 (+ 4998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:56:05.474996 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:56:05.475021 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:56:05.475481 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:56:05.475858 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:56:05.475861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:56:05.475863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:56:05.476092 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:56:05.476644 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:56:05.476652 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:56:05.477208 (+ 556us) client_negotiation.cc:769] Sending connection context
0925 22:56:05.478642 (+ 1434us) client_negotiation.cc:240] Negotiation successful
0925 22:56:05.479065 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1909,"thread_start_us":201,"threads_started":1}
I20260925 22:56:05.482465 32538 heartbeater.cc:347] Connected to a master server at 127.20.114.254:43839
I20260925 22:56:05.482755 32538 heartbeater.cc:464] Registering TS with master...
I20260925 22:56:05.483426 32538 heartbeater.cc:511] Master 127.20.114.254:43839 requested a full tablet report, sending...
I20260925 22:56:05.484841 32188 ts_manager.cc:194] Registered new tserver with Master: c4215cb3a29246a1b20ef6185d77dc83 (127.20.114.194:34937)
I20260925 22:56:05.485698 32188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:55869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
I20260925 22:56:05.518980 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.114.212:48297/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:43839
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:56:05 dist-test-slave-x1lh krb5kdc[31712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376965, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for HTTP/127.20.114.212@KRBTEST.COM
W20260925 22:56:05.715607 32543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:56:05.715998 32543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:56:05.716058 32543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:56:05.721645 32543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:56:05.721864 32543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:56:05.721987 32543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:56:05.727864 32543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.114.212:48297/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:43839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:56:05.729256 32543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:56:05.730719 32543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:56:05.740728 32549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:56:05.740738 32548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:56:05.741209 32551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:56:05.741690 32543 server_base.cc:1034] running on GCE node
I20260925 22:56:05.742142 32543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:56:05.742848 32543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:56:05.746210 32543 hybrid_clock.cc:648] HybridClock initialized: now 1790376965745961 us; error 263 us; skew 500 ppm
Sep 25 22:56:05 dist-test-slave-x1lh krb5kdc[31712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376965, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:56:05.750032 32543 init.cc:377] Logged in from keytab as kudu/127.20.114.195@KRBTEST.COM (short username kudu)
I20260925 22:56:05.751308 32543 webserver.cc:460] Webserver started at http://127.20.114.195:44093/ using document root <none> and password file <none>
I20260925 22:56:05.752113 32543 fs_manager.cc:359] Metadata directory not provided
I20260925 22:56:05.752207 32543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:56:05.752435 32543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:56:05 dist-test-slave-x1lh krb5kdc[31712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376965, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for HTTP/127.20.114.212@KRBTEST.COM
I20260925 22:56:05.771463 32543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "1399edca2111466c9b472396ed281b27"
format_stamp: "Formatted at 2026-09-25 22:56:05 on dist-test-slave-x1lh"
server_key: "14b759fef28102578611a0e37d499f3a"
server_key_iv: "e8835e411b548e30d8f2105389149f2b"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:05.772389 32543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "1399edca2111466c9b472396ed281b27"
format_stamp: "Formatted at 2026-09-25 22:56:05 on dist-test-slave-x1lh"
server_key: "14b759fef28102578611a0e37d499f3a"
server_key_iv: "e8835e411b548e30d8f2105389149f2b"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:05.783511 32543 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.004s sys 0.000s
I20260925 22:56:05.803581 32558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:05.805704 32543 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 22:56:05.805861 32543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "1399edca2111466c9b472396ed281b27"
format_stamp: "Formatted at 2026-09-25 22:56:05 on dist-test-slave-x1lh"
server_key: "14b759fef28102578611a0e37d499f3a"
server_key_iv: "e8835e411b548e30d8f2105389149f2b"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:05.806027 32543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:56:05.840044 32543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:56:05.844589 32543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:56:05.844803 32543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:56:05.845597 32543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:56:05.846776 32543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:56:05.846861 32543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:05.846931 32543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:56:05.847101 32543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:05.860988 32543 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:46167
I20260925 22:56:05.862313 32543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:56:05.862996 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 32543
I20260925 22:56:05.863116 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:56:05.863435 20939 external_mini_cluster.cc:1428] Setting key 3e9d73d4d8ab287dac3b8ac95763b510
I20260925 22:56:05.867029 32671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:46167 every 8 connection(s)
Sep 25 22:56:05 dist-test-slave-x1lh krb5kdc[31712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376965, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:56:05.887501 32404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:05.868044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:39411 (local address 127.20.114.254:43839)
0925 22:56:05.868180 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:56:05.868185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:56:05.869539 (+ 1354us) server_negotiation.cc:407] Connection header received
0925 22:56:05.870830 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:56:05.870834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:56:05.870897 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:56:05.871159 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:56:05.872786 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.873701 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.876124 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.876508 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.876602 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:05.881764 (+ 5162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:56:05.881769 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:56:05.881772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:56:05.881808 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:56:05.884473 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:05.885297 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:05.885301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:05.885303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:05.885408 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:05.885908 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:05.885912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:05.885914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:05.886196 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:56:05.886305 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 22:56:05.887154 (+ 849us) server_negotiation.cc:299] Negotiation successful
0925 22:56:05.887316 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 22:56:05.889798 32674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:05.866790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43839 (local address 127.20.114.195:39411)
0925 22:56:05.869306 (+ 2516us) negotiation.cc:107] Waiting for socket to connect
0925 22:56:05.869383 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0925 22:56:05.870514 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:56:05.871383 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:56:05.871394 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:56:05.871799 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:56:05.872578 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:05.872601 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.873893 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.873896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:05.875914 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:05.875930 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.877561 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.877564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:05.878122 (+ 558us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:05.878987 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:56:05.879013 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:56:05.881382 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:56:05.884661 (+ 3279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:56:05.884669 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:56:05.884692 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:56:05.885153 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:56:05.885542 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:56:05.885545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:56:05.885547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:56:05.885763 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:56:05.886330 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:56:05.886339 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:56:05.886736 (+ 397us) client_negotiation.cc:769] Sending connection context
0925 22:56:05.887774 (+ 1038us) client_negotiation.cc:240] Negotiation successful
0925 22:56:05.888184 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2242,"thread_start_us":174,"threads_started":1}
I20260925 22:56:05.891407 32672 heartbeater.cc:347] Connected to a master server at 127.20.114.254:43839
I20260925 22:56:05.891707 32672 heartbeater.cc:464] Registering TS with master...
I20260925 22:56:05.892614 32672 heartbeater.cc:511] Master 127.20.114.254:43839 requested a full tablet report, sending...
I20260925 22:56:05.894179 32188 ts_manager.cc:194] Registered new tserver with Master: 1399edca2111466c9b472396ed281b27 (127.20.114.195:46167)
I20260925 22:56:05.894898 32188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:39411
I20260925 22:56:05.905212 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:56:05.918172 32404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:05.908227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49332 (local address 127.20.114.254:43839)
0925 22:56:05.908357 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:56:05.908362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:56:05.909002 (+ 640us) server_negotiation.cc:407] Connection header received
0925 22:56:05.909231 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:56:05.909235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:56:05.909290 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:56:05.909346 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:56:05.910416 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.910948 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.912398 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.912638 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.912768 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:05.914610 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:56:05.914613 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:56:05.914616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:56:05.914643 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:56:05.916235 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:05.916737 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:05.916742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:05.916744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:05.916838 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:05.917188 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:05.917192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:05.917194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:05.917435 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:56:05.917531 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 22:56:05.917852 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 22:56:05.917983 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:56:05.921939 32188 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:56:05.926550 32188 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 25 22:56:05 dist-test-slave-x1lh krb5kdc[31712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376965, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for HTTP/127.20.114.212@KRBTEST.COM
I20260925 22:56:05.968695 32686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:05.960006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40530 (local address 127.20.114.194:34937)
0925 22:56:05.961557 (+ 1551us) server_negotiation.cc:206] Beginning negotiation
0925 22:56:05.961576 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:56:05.961607 (+ 31us) server_negotiation.cc:407] Connection header received
0925 22:56:05.961699 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:56:05.961706 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:56:05.961955 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:56:05.962145 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:56:05.962512 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.964204 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.966939 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.967872 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.968014 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:05.968184 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 22:56:05.968424 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 22:56:05.968540 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1326,"thread_start_us":158,"threads_started":1}
I20260925 22:56:05.968688 32685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:05.959038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:34937 (local address 127.0.0.1:40530)
0925 22:56:05.959543 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 22:56:05.959606 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:56:05.959754 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:56:05.962154 (+ 2400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:56:05.962162 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:56:05.962189 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:56:05.962372 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:05.962384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.964348 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.964352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:05.965888 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:05.965899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.968036 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.968040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:05.968187 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:05.968237 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 22:56:05.968369 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 22:56:05.968512 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":136,"threads_started":1}
I20260925 22:56:05.972554 32470 tablet_service.cc:1467] Processing CreateTablet for tablet e61a4056c4e844ccbf11a52c30111376 (DEFAULT_TABLE table=test-table [id=4f436b3fec364df1921b15fdf0948595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:56:05.973866 32470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61a4056c4e844ccbf11a52c30111376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:56:05.980581 32688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:05.961105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:46167 (local address 127.0.0.1:52666)
0925 22:56:05.962403 (+ 1298us) negotiation.cc:107] Waiting for socket to connect
0925 22:56:05.962418 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:56:05.962510 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:56:05.963218 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:56:05.963221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:56:05.963239 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:56:05.963348 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:05.963354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.964646 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.964650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:05.969449 (+ 4799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:05.969461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.979784 (+ 10323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.979788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:05.980211 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:05.980240 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 22:56:05.980322 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 22:56:05.980409 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1229,"mutex_wait_us":276,"thread_start_us":130,"threads_started":1}
I20260925 22:56:05.981766 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:05.961591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52666 (local address 127.20.114.195:46167)
0925 22:56:05.961962 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0925 22:56:05.961975 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:56:05.962628 (+ 653us) server_negotiation.cc:407] Connection header received
0925 22:56:05.962714 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:56:05.962719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:56:05.962923 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:56:05.963074 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:56:05.963481 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.964530 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.969637 (+ 5107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.979444 (+ 9807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.979576 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:05.979713 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 22:56:05.981483 (+ 1770us) server_negotiation.cc:299] Negotiation successful
0925 22:56:05.981608 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":191,"threads_started":1}
I20260925 22:56:05.985554 32599 tablet_service.cc:1467] Processing CreateTablet for tablet e61a4056c4e844ccbf11a52c30111376 (DEFAULT_TABLE table=test-table [id=4f436b3fec364df1921b15fdf0948595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:56:05.986647 32599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61a4056c4e844ccbf11a52c30111376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:56:05.987479 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:05.961762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41292 (local address 127.20.114.193:43893)
0925 22:56:05.973211 (+ 11449us) server_negotiation.cc:206] Beginning negotiation
0925 22:56:05.973219 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:56:05.977022 (+ 3803us) server_negotiation.cc:407] Connection header received
0925 22:56:05.977271 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:56:05.977278 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:56:05.977495 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:56:05.977638 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:56:05.978096 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.979281 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.982145 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.983002 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.983106 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:05.983232 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 22:56:05.987184 (+ 3952us) server_negotiation.cc:299] Negotiation successful
0925 22:56:05.987322 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11310,"thread_start_us":88,"threads_started":1}
I20260925 22:56:05.987979 32685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:05.961336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:43893 (local address 127.0.0.1:41292)
0925 22:56:05.969179 (+ 7843us) negotiation.cc:107] Waiting for socket to connect
0925 22:56:05.976912 (+ 7733us) client_negotiation.cc:174] Beginning negotiation
0925 22:56:05.977114 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:56:05.977838 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:56:05.977841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:56:05.977860 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:56:05.977953 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:05.977959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.979456 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.979459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:05.981989 (+ 2530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:05.982001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:05.983542 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:05.983546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:05.983642 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:05.983667 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:56:05.987727 (+ 4060us) client_negotiation.cc:240] Negotiation successful
0925 22:56:05.987834 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7729,"mutex_wait_us":1,"thread_start_us":110,"threads_started":1}
I20260925 22:56:05.991782 32691 tablet_bootstrap.cc:492] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83: Bootstrap starting.
I20260925 22:56:05.991436 32325 tablet_service.cc:1467] Processing CreateTablet for tablet e61a4056c4e844ccbf11a52c30111376 (DEFAULT_TABLE table=test-table [id=4f436b3fec364df1921b15fdf0948595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:56:05.992882 32325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61a4056c4e844ccbf11a52c30111376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:56:05.994215 32691 tablet_bootstrap.cc:654] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83: Neither blocks nor log segments found. Creating new log.
I20260925 22:56:05.995795 32691 log.cc:824] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83: Log is configured to *not* fsync() on all Append() calls
I20260925 22:56:05.998014 32691 tablet_bootstrap.cc:492] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83: No bootstrap required, opened a new log
I20260925 22:56:05.998242 32691 ts_tablet_manager.cc:1397] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:56:06.002377 32691 raft_consensus.cc:348] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4215cb3a29246a1b20ef6185d77dc83" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 34937 } } peers { permanent_uuid: "1399edca2111466c9b472396ed281b27" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 46167 } } peers { permanent_uuid: "f141321b518941c79548115073ae6ac5" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 43893 } }
I20260925 22:56:06.002704 32691 raft_consensus.cc:374] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:56:06.002761 32691 raft_consensus.cc:729] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4215cb3a29246a1b20ef6185d77dc83, State: Initialized, Role: FOLLOWER
I20260925 22:56:06.003319 32691 consensus_queue.cc:260] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4215cb3a29246a1b20ef6185d77dc83" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 34937 } } peers { permanent_uuid: "1399edca2111466c9b472396ed281b27" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 46167 } } peers { permanent_uuid: "f141321b518941c79548115073ae6ac5" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 43893 } }
I20260925 22:56:06.004766 32691 ts_tablet_manager.cc:1428] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:56:06.005584 32692 tablet_bootstrap.cc:492] T e61a4056c4e844ccbf11a52c30111376 P 1399edca2111466c9b472396ed281b27: Bootstrap starting.
I20260925 22:56:06.008802 32692 tablet_bootstrap.cc:654] T e61a4056c4e844ccbf11a52c30111376 P 1399edca2111466c9b472396ed281b27: Neither blocks nor log segments found. Creating new log.
I20260925 22:56:06.009475 32694 tablet_bootstrap.cc:492] T e61a4056c4e844ccbf11a52c30111376 P f141321b518941c79548115073ae6ac5: Bootstrap starting.
I20260925 22:56:06.009918 32692 log.cc:824] T e61a4056c4e844ccbf11a52c30111376 P 1399edca2111466c9b472396ed281b27: Log is configured to *not* fsync() on all Append() calls
I20260925 22:56:06.010649 32538 heartbeater.cc:503] Master 127.20.114.254:43839 was elected leader, sending a full tablet report...
I20260925 22:56:06.015374 32694 tablet_bootstrap.cc:654] T e61a4056c4e844ccbf11a52c30111376 P f141321b518941c79548115073ae6ac5: Neither blocks nor log segments found. Creating new log.
I20260925 22:56:06.016367 32694 log.cc:824] T e61a4056c4e844ccbf11a52c30111376 P f141321b518941c79548115073ae6ac5: Log is configured to *not* fsync() on all Append() calls
I20260925 22:56:06.016788 32692 tablet_bootstrap.cc:492] T e61a4056c4e844ccbf11a52c30111376 P 1399edca2111466c9b472396ed281b27: No bootstrap required, opened a new log
I20260925 22:56:06.017058 32692 ts_tablet_manager.cc:1397] T e61a4056c4e844ccbf11a52c30111376 P 1399edca2111466c9b472396ed281b27: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260925 22:56:06.018824 32694 tablet_bootstrap.cc:492] T e61a4056c4e844ccbf11a52c30111376 P f141321b518941c79548115073ae6ac5: No bootstrap required, opened a new log
I20260925 22:56:06.019021 32694 ts_tablet_manager.cc:1397] T e61a4056c4e844ccbf11a52c30111376 P f141321b518941c79548115073ae6ac5: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260925 22:56:06.022927 32692 raft_consensus.cc:348] T e61a4056c4e844ccbf11a52c30111376 P 1399edca2111466c9b472396ed281b27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4215cb3a29246a1b20ef6185d77dc83" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 34937 } } peers { permanent_uuid: "1399edca2111466c9b472396ed281b27" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 46167 } } peers { permanent_uuid: "f141321b518941c79548115073ae6ac5" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 43893 } }
I20260925 22:56:06.023275 32692 raft_consensus.cc:374] T e61a4056c4e844ccbf11a52c30111376 P 1399edca2111466c9b472396ed281b27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:56:06.023347 32692 raft_consensus.cc:729] T e61a4056c4e844ccbf11a52c30111376 P 1399edca2111466c9b472396ed281b27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1399edca2111466c9b472396ed281b27, State: Initialized, Role: FOLLOWER
I20260925 22:56:06.024052 32692 consensus_queue.cc:260] T e61a4056c4e844ccbf11a52c30111376 P 1399edca2111466c9b472396ed281b27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4215cb3a29246a1b20ef6185d77dc83" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 34937 } } peers { permanent_uuid: "1399edca2111466c9b472396ed281b27" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 46167 } } peers { permanent_uuid: "f141321b518941c79548115073ae6ac5" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 43893 } }
I20260925 22:56:06.025066 32672 heartbeater.cc:503] Master 127.20.114.254:43839 was elected leader, sending a full tablet report...
I20260925 22:56:06.025592 32692 ts_tablet_manager.cc:1428] T e61a4056c4e844ccbf11a52c30111376 P 1399edca2111466c9b472396ed281b27: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260925 22:56:06.028119 32694 raft_consensus.cc:348] T e61a4056c4e844ccbf11a52c30111376 P f141321b518941c79548115073ae6ac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4215cb3a29246a1b20ef6185d77dc83" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 34937 } } peers { permanent_uuid: "1399edca2111466c9b472396ed281b27" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 46167 } } peers { permanent_uuid: "f141321b518941c79548115073ae6ac5" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 43893 } }
I20260925 22:56:06.028563 32694 raft_consensus.cc:374] T e61a4056c4e844ccbf11a52c30111376 P f141321b518941c79548115073ae6ac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:56:06.028680 32694 raft_consensus.cc:729] T e61a4056c4e844ccbf11a52c30111376 P f141321b518941c79548115073ae6ac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f141321b518941c79548115073ae6ac5, State: Initialized, Role: FOLLOWER
I20260925 22:56:06.029976 32694 consensus_queue.cc:260] T e61a4056c4e844ccbf11a52c30111376 P f141321b518941c79548115073ae6ac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4215cb3a29246a1b20ef6185d77dc83" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 34937 } } peers { permanent_uuid: "1399edca2111466c9b472396ed281b27" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 46167 } } peers { permanent_uuid: "f141321b518941c79548115073ae6ac5" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 43893 } }
I20260925 22:56:06.030759 32402 heartbeater.cc:503] Master 127.20.114.254:43839 was elected leader, sending a full tablet report...
I20260925 22:56:06.031533 32694 ts_tablet_manager.cc:1428] T e61a4056c4e844ccbf11a52c30111376 P f141321b518941c79548115073ae6ac5: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
W20260925 22:56:06.052521 32403 tablet.cc:2369] T e61a4056c4e844ccbf11a52c30111376 P f141321b518941c79548115073ae6ac5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:56:06.117797 32673 tablet.cc:2369] T e61a4056c4e844ccbf11a52c30111376 P 1399edca2111466c9b472396ed281b27: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:56:06.205655 32539 tablet.cc:2369] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:56:06.342825 32695 raft_consensus.cc:482] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:56:06.343124 32695 raft_consensus.cc:504] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4215cb3a29246a1b20ef6185d77dc83" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 34937 } } peers { permanent_uuid: "1399edca2111466c9b472396ed281b27" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 46167 } } peers { permanent_uuid: "f141321b518941c79548115073ae6ac5" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 43893 } }
I20260925 22:56:06.344668 32695 leader_election.cc:288] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1399edca2111466c9b472396ed281b27 (127.20.114.195:46167), f141321b518941c79548115073ae6ac5 (127.20.114.193:43893)
I20260925 22:56:06.350575 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:06.345450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:48341 (local address 127.20.114.195:46167)
0925 22:56:06.345594 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:56:06.345599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:56:06.345851 (+ 252us) server_negotiation.cc:407] Connection header received
0925 22:56:06.346058 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:56:06.346062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:56:06.346111 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:56:06.346162 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:56:06.346579 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:06.347336 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:06.349111 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:06.349724 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:06.349829 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:06.349884 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:56:06.350337 (+ 453us) server_negotiation.cc:299] Negotiation successful
0925 22:56:06.350425 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:56:06.350570 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:06.345461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:37827 (local address 127.20.114.193:43893)
0925 22:56:06.345592 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:56:06.345597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:56:06.346016 (+ 419us) server_negotiation.cc:407] Connection header received
0925 22:56:06.346214 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:56:06.346218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:56:06.346269 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:56:06.346316 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:56:06.346762 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:06.347518 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:06.349131 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:06.349746 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:06.349846 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:06.349897 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 22:56:06.350336 (+ 439us) server_negotiation.cc:299] Negotiation successful
0925 22:56:06.350431 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:56:06.351128 32708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:06.345003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:46167 (local address 127.20.114.194:48341)
0925 22:56:06.345757 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0925 22:56:06.345772 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:56:06.345934 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:56:06.346301 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:56:06.346304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:56:06.346320 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:56:06.346446 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:06.346452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:06.347497 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:06.347500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:06.348927 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:06.348938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:06.350008 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:06.350011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:06.350118 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:06.350175 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 22:56:06.350874 (+ 699us) client_negotiation.cc:240] Negotiation successful
0925 22:56:06.350962 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":77,"thread_start_us":126,"threads_started":1}
I20260925 22:56:06.351562 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:06.345067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:43893 (local address 127.20.114.194:37827)
0925 22:56:06.345914 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0925 22:56:06.345930 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:56:06.346091 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:56:06.346454 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:56:06.346457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:56:06.346473 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:56:06.346583 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:06.346631 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:06.347680 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:06.347683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:06.348981 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:06.348993 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:06.350030 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:06.350034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:06.350189 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:06.350213 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:56:06.351315 (+ 1102us) client_negotiation.cc:240] Negotiation successful
0925 22:56:06.351399 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":40,"thread_start_us":109,"threads_started":1}
I20260925 22:56:06.351845 32620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61a4056c4e844ccbf11a52c30111376" candidate_uuid: "c4215cb3a29246a1b20ef6185d77dc83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1399edca2111466c9b472396ed281b27" is_pre_election: true
I20260925 22:56:06.352210 32620 raft_consensus.cc:2454] T e61a4056c4e844ccbf11a52c30111376 P 1399edca2111466c9b472396ed281b27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4215cb3a29246a1b20ef6185d77dc83 in term 0.
I20260925 22:56:06.352545 32355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61a4056c4e844ccbf11a52c30111376" candidate_uuid: "c4215cb3a29246a1b20ef6185d77dc83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f141321b518941c79548115073ae6ac5" is_pre_election: true
I20260925 22:56:06.352793 32426 leader_election.cc:302] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1399edca2111466c9b472396ed281b27, c4215cb3a29246a1b20ef6185d77dc83; no voters:
I20260925 22:56:06.352922 32355 raft_consensus.cc:2454] T e61a4056c4e844ccbf11a52c30111376 P f141321b518941c79548115073ae6ac5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4215cb3a29246a1b20ef6185d77dc83 in term 0.
I20260925 22:56:06.353235 32695 raft_consensus.cc:2781] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:56:06.353323 32695 raft_consensus.cc:482] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:56:06.353363 32695 raft_consensus.cc:3037] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:56:06.354736 32695 raft_consensus.cc:504] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4215cb3a29246a1b20ef6185d77dc83" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 34937 } } peers { permanent_uuid: "1399edca2111466c9b472396ed281b27" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 46167 } } peers { permanent_uuid: "f141321b518941c79548115073ae6ac5" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 43893 } }
I20260925 22:56:06.355142 32695 leader_election.cc:288] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83 [CANDIDATE]: Term 1 election: Requested vote from peers 1399edca2111466c9b472396ed281b27 (127.20.114.195:46167), f141321b518941c79548115073ae6ac5 (127.20.114.193:43893)
I20260925 22:56:06.355603 32355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61a4056c4e844ccbf11a52c30111376" candidate_uuid: "c4215cb3a29246a1b20ef6185d77dc83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f141321b518941c79548115073ae6ac5"
I20260925 22:56:06.355644 32620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61a4056c4e844ccbf11a52c30111376" candidate_uuid: "c4215cb3a29246a1b20ef6185d77dc83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1399edca2111466c9b472396ed281b27"
I20260925 22:56:06.355733 32355 raft_consensus.cc:3037] T e61a4056c4e844ccbf11a52c30111376 P f141321b518941c79548115073ae6ac5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:56:06.355775 32620 raft_consensus.cc:3037] T e61a4056c4e844ccbf11a52c30111376 P 1399edca2111466c9b472396ed281b27 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:56:06.357056 32355 raft_consensus.cc:2454] T e61a4056c4e844ccbf11a52c30111376 P f141321b518941c79548115073ae6ac5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4215cb3a29246a1b20ef6185d77dc83 in term 1.
I20260925 22:56:06.357076 32620 raft_consensus.cc:2454] T e61a4056c4e844ccbf11a52c30111376 P 1399edca2111466c9b472396ed281b27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4215cb3a29246a1b20ef6185d77dc83 in term 1.
I20260925 22:56:06.357460 32426 leader_election.cc:302] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1399edca2111466c9b472396ed281b27, c4215cb3a29246a1b20ef6185d77dc83; no voters:
I20260925 22:56:06.357669 32695 raft_consensus.cc:2781] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:56:06.357895 32695 raft_consensus.cc:686] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83 [term 1 LEADER]: Becoming Leader. State: Replica: c4215cb3a29246a1b20ef6185d77dc83, State: Running, Role: LEADER
I20260925 22:56:06.358274 32695 consensus_queue.cc:237] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4215cb3a29246a1b20ef6185d77dc83" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 34937 } } peers { permanent_uuid: "1399edca2111466c9b472396ed281b27" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 46167 } } peers { permanent_uuid: "f141321b518941c79548115073ae6ac5" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 43893 } }
I20260925 22:56:06.362352 32188 catalog_manager.cc:5495] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4215cb3a29246a1b20ef6185d77dc83 (127.20.114.194). New cstate: current_term: 1 leader_uuid: "c4215cb3a29246a1b20ef6185d77dc83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4215cb3a29246a1b20ef6185d77dc83" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 34937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1399edca2111466c9b472396ed281b27" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 46167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f141321b518941c79548115073ae6ac5" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 43893 } health_report { overall_health: UNKNOWN } } }
I20260925 22:56:06.425644 32686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:06.421088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40544 (local address 127.20.114.194:34937)
0925 22:56:06.421210 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 22:56:06.421215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:56:06.421430 (+ 215us) server_negotiation.cc:407] Connection header received
0925 22:56:06.421613 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:56:06.421617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:56:06.421694 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:56:06.421812 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:56:06.422084 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:06.422866 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:06.423789 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:06.424133 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:06.424242 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:06.424600 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:56:06.425216 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:56:06.425336 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 22:56:06.425416 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 22:56:06.425483 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 22:56:06.432853 32620 raft_consensus.cc:1261] T e61a4056c4e844ccbf11a52c30111376 P 1399edca2111466c9b472396ed281b27 [term 1 FOLLOWER]: Refusing update from remote peer c4215cb3a29246a1b20ef6185d77dc83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:56:06.432899 32355 raft_consensus.cc:1261] T e61a4056c4e844ccbf11a52c30111376 P f141321b518941c79548115073ae6ac5 [term 1 FOLLOWER]: Refusing update from remote peer c4215cb3a29246a1b20ef6185d77dc83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:56:06.433761 32710 consensus_queue.cc:1035] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f141321b518941c79548115073ae6ac5" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:56:06.434142 32695 consensus_queue.cc:1035] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1399edca2111466c9b472396ed281b27" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 46167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:56:06.443693 32714 mvcc.cc:205] Tried to move back new op lower bound from 7333384054501203968 to 7333384054206844928. Current Snapshot: MvccSnapshot[applied={T|T < 7333384054501203968}]
I20260925 22:56:06.445706 32715 mvcc.cc:205] Tried to move back new op lower bound from 7333384054501203968 to 7333384054206844928. Current Snapshot: MvccSnapshot[applied={T|T < 7333384054501203968}]
I20260925 22:56:06.455850 32187 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:56:06.456069 32187 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:56:06.459122 32187 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6429010135143fcbde03114b4d64b23: Sending DeleteTablet for 3 replicas of tablet e61a4056c4e844ccbf11a52c30111376
I20260925 22:56:06.460036 32470 tablet_service.cc:1514] Processing DeleteTablet for tablet e61a4056c4e844ccbf11a52c30111376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:56:06 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:40530
I20260925 22:56:06.461148 32599 tablet_service.cc:1514] Processing DeleteTablet for tablet e61a4056c4e844ccbf11a52c30111376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:56:06 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:52666
I20260925 22:56:06.462147 32325 tablet_service.cc:1514] Processing DeleteTablet for tablet e61a4056c4e844ccbf11a52c30111376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:56:06 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:41292
I20260925 22:56:06.463701 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 32269
I20260925 22:56:06.464789 32723 tablet_replica.cc:330] T e61a4056c4e844ccbf11a52c30111376 P 1399edca2111466c9b472396ed281b27: stopping tablet replica
I20260925 22:56:06.465279 32723 raft_consensus.cc:2229] T e61a4056c4e844ccbf11a52c30111376 P 1399edca2111466c9b472396ed281b27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:56:06.465607 32722 tablet_replica.cc:330] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83: stopping tablet replica
I20260925 22:56:06.465986 32722 raft_consensus.cc:2229] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:56:06.466404 32723 raft_consensus.cc:2258] T e61a4056c4e844ccbf11a52c30111376 P 1399edca2111466c9b472396ed281b27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:56:06.466884 32722 raft_consensus.cc:2258] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:56:06.469043 32722 ts_tablet_manager.cc:1905] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:56:06.472303 32722 ts_tablet_manager.cc:1918] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:56:06.472425 32722 log.cc:1196] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/e61a4056c4e844ccbf11a52c30111376
I20260925 22:56:06.472795 32722 ts_tablet_manager.cc:1939] T e61a4056c4e844ccbf11a52c30111376 P c4215cb3a29246a1b20ef6185d77dc83: Deleting consensus metadata
W20260925 22:56:06.474221 32175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.114.193:43893 (error 108)
I20260925 22:56:06.474906 32174 catalog_manager.cc:4841] TS c4215cb3a29246a1b20ef6185d77dc83 (127.20.114.194:34937): tablet e61a4056c4e844ccbf11a52c30111376 (table test-table [id=4f436b3fec364df1921b15fdf0948595]) successfully deleted
W20260925 22:56:06.475577 32685 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:56:06.475272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:43893 (local address 127.0.0.1:41300)
0925 22:56:06.475393 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 22:56:06.475477 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.193:43893: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 22:56:06.475783 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 32408
W20260925 22:56:06.475822 32175 catalog_manager.cc:4584] TS f141321b518941c79548115073ae6ac5 (127.20.114.193:43893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e61a4056c4e844ccbf11a52c30111376: Network error: Client connection negotiation failed: client connection to 127.20.114.193:43893: connect: Connection reset by peer (error 104)
I20260925 22:56:06.482071 32723 ts_tablet_manager.cc:1905] T e61a4056c4e844ccbf11a52c30111376 P 1399edca2111466c9b472396ed281b27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:56:06.484678 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 32543
I20260925 22:56:06.491891 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 32156
I20260925 22:56:06.500437 20939 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 22:56:06.846819 20939 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 22:56:06.846947 20939 mini_ranger.cc:66] Stopping Ranger...
I20260925 22:56:07.193004 20939 mini_ranger.cc:68] Stopped Ranger
2026-09-25 22:56:07.193 UTC [31742] LOG: received smart shutdown request
2026-09-25 22:56:07.197 UTC [31742] LOG: background worker "logical replication launcher" (PID 31751) exited with exit code 1
2026-09-25 22:56:07.197 UTC [31746] LOG: shutting down
2026-09-25 22:56:07.397 UTC [31742] LOG: database system is shut down
2026-09-25T22:56:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48651 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:56:07 dist-test-slave-x1lh krb5kdc[32734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:56:07 dist-test-slave-x1lh krb5kdc[32734](info): set up 2 sockets
Sep 25 22:56:07 dist-test-slave-x1lh krb5kdc[32734](info): commencing operation
krb5kdc: starting...
W20260925 22:56:08.573228 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:56:08 dist-test-slave-x1lh krb5kdc[32734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:56:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:56:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.114.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.114.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.114.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/rangeradmin_127.20.114.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.114.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/rangeradmin_127.20.114.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.114.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.114.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.114.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/rangerlookup_127.20.114.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.114.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/rangerlookup_127.20.114.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.114.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.114.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.114.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/HTTP_127.20.114.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.114.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/HTTP_127.20.114.212@KRBTEST.COM.keytab.
I20260925 22:56:08.765218 20939 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskV8JYeA/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/postgres -l logfile start
2026-09-25 22:56:11.590 UTC [32764] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 22:56:11.590 UTC [32764] LOG: listening on IPv4 address "127.20.114.212", port 35629
2026-09-25 22:56:11.594 UTC [32764] LOG: listening on Unix socket "/tmp/.s.PGSQL.35629"
2026-09-25 22:56:11.616 UTC [32767] LOG: database system was shut down at 2026-09-25 22:56:09 UTC
2026-09-25 22:56:11.620 UTC [32764] LOG: database system is ready to accept connections
I20260925 22:56:12.612994 20939 mini_postgres.cc:96] Postgres bound to 35629
2026-09-25 22:56:12.619 UTC [307] FATAL: database "slave" does not exist
127.20.114.212:35629 - accepting connections
I20260925 22:56:12.620270 20939 mini_ranger.cc:161] Starting Ranger...
I20260925 22:56:12.637012 20939 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 22:56:13.289044 20939 mini_ranger.cc:87] Created ranger Postgres database
I20260925 22:56:13.289146 20939 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ranger-admin
2026-09-25 22:56:13,588 [I] DB FLAVOR :POSTGRES
2026-09-25 22:56:13,589 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:56:13,589 [I] Checking connection..
2026-09-25 22:56:13,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:35629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 22:56:13,868 [I] Checking connection passed.
2026-09-25 22:56:13,869 [I] --------- Verifying version history table ---------
2026-09-25 22:56:13,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:35629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:56:14,137 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:56:14,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:35629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:56:14,405 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:56:14,405 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 22:56:14,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:35629/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 22:56:14.672 UTC [371] WARNING: there is no transaction in progress
2026-09-25 22:56:14,680 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 22:56:14,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:35629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:56:14,947 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 22:56:14,947 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:56:14,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:35629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 22:56:15,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:35629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 22:56:15,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:35629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-x1lh.c.gcp-upstream.internal','N') ;"
2026-09-25 22:56:15,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:35629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:56:15,915 [I] Table x_portal_user does not exist in database ranger
2026-09-25 22:56:15,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:35629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:56:16,157 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 22:56:16,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:35629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:56:16,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:35629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:56:16,685 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 22:56:16,686 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:35629/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 22:56:17.781 UTC [479] WARNING: there is no transaction in progress
2026-09-25 22:56:17.801 UTC [479] WARNING: there is no transaction in progress
2026-09-25 22:56:17.822 UTC [479] WARNING: there is no transaction in progress
2026-09-25 22:56:17.842 UTC [479] WARNING: there is no transaction in progress
2026-09-25 22:56:17.862 UTC [479] WARNING: there is no transaction in progress
2026-09-25 22:56:17.882 UTC [479] WARNING: there is no transaction in progress
2026-09-25 22:56:17.948 UTC [479] WARNING: there is no transaction in progress
2026-09-25 22:56:17.956 UTC [479] WARNING: there is no transaction in progress
2026-09-25 22:56:17.973 UTC [479] WARNING: there is no transaction in progress
2026-09-25 22:56:17.986 UTC [479] WARNING: there is no transaction in progress
2026-09-25 22:56:17.998 UTC [479] WARNING: there is no transaction in progress
2026-09-25 22:56:18.015 UTC [479] WARNING: there is no transaction in progress
2026-09-25 22:56:18.027 UTC [479] WARNING: there is no transaction in progress
2026-09-25 22:56:18.036 UTC [479] WARNING: there is no transaction in progress
2026-09-25 22:56:18.045 UTC [479] WARNING: there is no transaction in progress
2026-09-25 22:56:18.055 UTC [479] WARNING: there is no transaction in progress
2026-09-25 22:56:18,931 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 22:56:18,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:35629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 22:56:19,177 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 22:56:19,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:35629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:56:19,435 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 22:56:19,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:35629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:56:19,699 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 22:56:19,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:35629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:56:19,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:35629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:56:20,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:35629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-x1lh.c.gcp-upstream.internal';"
2026-09-25 22:56:20,479 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 22:56:20,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:35629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 22:56:20,734 [I] DB_PATCHES have already been applied
I20260925 22:56:20.740260 20939 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ranger-admin:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskV8JYeA/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 22:56:20.740361 20939 mini_ranger.cc:193] Using host: 127.20.114.212
I20260925 22:56:20.744051 20939 mini_ranger.cc:227] Ranger admin URL: http://127.20.114.212:48687
Sep 25, 2026 10:56:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:56:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:56:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:56:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:56:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:56:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:56:21 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 22:56:21.928368 20939 mini_ranger.cc:160] Time spent starting Ranger: real 9.308s user 0.000s sys 0.011s
Sep 25, 2026 10:56:44 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:56:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 10:56:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:56:44 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:56:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:56:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:56:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:56:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:56:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:56:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:56:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:56:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:56:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:56:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:56:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:56:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:56:44 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:56:45 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 22:56:46.437924 20939 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.114.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.114.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.114.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/rangerkms_127.20.114.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.114.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/rangerkms_127.20.114.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.114.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.114.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.114.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/HTTP_127.20.114.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.114.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/HTTP_127.20.114.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 22:56:46 dist-test-slave-x1lh krb5kdc[32734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377006, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 22:56:46.544325 20939 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 22:56:46.555204 20939 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 22:56:47.735483 20939 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 22:56:47.735589 20939 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ranger-kms
I20260925 22:56:47.735611 20939 mini_ranger_kms.cc:227] Using postgres at 127.20.114.212:35629
2026-09-25 22:56:47,925 [I] DB FLAVOR :POSTGRES
2026-09-25 22:56:47,925 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:56:47,925 [I] Checking connection
2026-09-25 22:56:47,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:35629/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 22:56:48,192 [I] connection success
2026-09-25 22:56:48,193 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 22:56:48,193 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 22:56:48,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:35629/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 22:56:48,448 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 22:56:48,448 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:56:48,448 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 22:56:48,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.114.212:35629/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 22:56:48,753 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 22:56:49.333401 20939 mini_ranger_kms.cc:310] Created kms service
I20260925 22:56:49.578651 20939 mini_ranger_kms.cc:326] Created kudu user
I20260925 22:56:49.631623 20939 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 22:56:49 dist-test-slave-x1lh krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377006, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.114.212@KRBTEST.COM
I20260925 22:56:50.142450 20939 mini_ranger_kms.cc:384] Added ranger policy
I20260925 22:56:50.142607 20939 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ranger-kms:/tmp/dist-test-taskV8JYeA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskV8JYeA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 22:56:50.142628 20939 mini_ranger_kms.cc:240] Using host: 127.20.114.212
I20260925 22:56:50.146689 20939 mini_ranger_kms.cc:276] Ranger KMS PID: 660
I20260925 22:56:50.146788 20939 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.114.212:52321
Sep 25, 2026 10:56:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:56:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:56:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:56:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:56:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:56:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 22:56:52 dist-test-slave-x1lh krb5kdc[32734](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790377012, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.114.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:56:52.500022 20939 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.956s user 0.004s sys 0.169s
I20260925 22:56:52.500172 20939 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 22:56:52.500239 20939 mini_ranger_kms.cc:401] 127.20.114.212:52321/kms/v1/keys
Sep 25 22:56:52 dist-test-slave-x1lh krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377012, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.114.212@KRBTEST.COM for HTTP/127.20.114.212@KRBTEST.COM
Sep 25 22:56:53 dist-test-slave-x1lh krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377012, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.114.212@KRBTEST.COM for HTTP/127.20.114.212@KRBTEST.COM
Sep 25 22:56:55 dist-test-slave-x1lh krb5kdc[32734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:56:55.729681 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:45753
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:44771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:45753
--ranger_config_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.114.212:52321/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:56:55.863947 710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:56:55.864276 710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:56:55.864320 710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:56:55.870827 710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:56:55.870923 710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:56:55.870944 710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:56:55.870962 710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:56:55.870978 710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:56:55.879051 710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:44771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.114.212:52321/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:45753
--ranger_config_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:45753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:56:55.880303 710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:56:55.881575 710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:56:55.892393 710 server_base.cc:1034] running on GCE node
W20260925 22:56:55.892401 715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:56:55.892649 716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:56:55.892647 718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:56:55.893695 710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:56:55.895035 710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:56:55.896302 710 hybrid_clock.cc:648] HybridClock initialized: now 1790377015896265 us; error 66 us; skew 500 ppm
Sep 25 22:56:55 dist-test-slave-x1lh krb5kdc[32734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377015, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:56:55.899576 710 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:56:55.900781 710 webserver.cc:460] Webserver started at http://127.20.114.254:45539/ using document root <none> and password file <none>
I20260925 22:56:55.901427 710 fs_manager.cc:359] Metadata directory not provided
I20260925 22:56:55.901527 710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:56:55.901814 710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:56:55 dist-test-slave-x1lh krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377015, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for HTTP/127.20.114.212@KRBTEST.COM
I20260925 22:56:55.971202 710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "f833443ce16648f08892528119e66287"
format_stamp: "Formatted at 2026-09-25 22:56:55 on dist-test-slave-x1lh"
server_key: "e568926063aeeb5968b3b0868367392b"
server_key_iv: "193937e34af6102029b7ccf16351648e"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:55.971977 710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "f833443ce16648f08892528119e66287"
format_stamp: "Formatted at 2026-09-25 22:56:55 on dist-test-slave-x1lh"
server_key: "e568926063aeeb5968b3b0868367392b"
server_key_iv: "193937e34af6102029b7ccf16351648e"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:55.976601 710 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 22:56:55.997004 725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:55.998708 710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:56:55.998850 710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "f833443ce16648f08892528119e66287"
format_stamp: "Formatted at 2026-09-25 22:56:55 on dist-test-slave-x1lh"
server_key: "e568926063aeeb5968b3b0868367392b"
server_key_iv: "193937e34af6102029b7ccf16351648e"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:55.999030 710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:56:56.031549 710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:56:56.035457 710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:56:56.035703 710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:56:56.045431 710 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:45753
I20260925 22:56:56.045431 777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:45753 every 8 connection(s)
I20260925 22:56:56.046770 710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:56:56.049732 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 710
I20260925 22:56:56.049922 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:56:56.050226 20939 external_mini_cluster.cc:1428] Setting key cf42b84a4984c17342999aaca94d1301
I20260925 22:56:56.051304 778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:56:56.059571 778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287: Bootstrap starting.
Sep 25 22:56:56 dist-test-slave-x1lh krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:56:56.063179 778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287: Neither blocks nor log segments found. Creating new log.
I20260925 22:56:56.064810 778 log.cc:824] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287: Log is configured to *not* fsync() on all Append() calls
I20260925 22:56:56.067164 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:56.052157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35176 (local address 127.20.114.254:45753)
0925 22:56:56.052651 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0925 22:56:56.052673 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 22:56:56.052727 (+ 54us) server_negotiation.cc:407] Connection header received
0925 22:56:56.053502 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:56:56.053523 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:56:56.053925 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:56:56.054375 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:56:56.055302 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:56.056369 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:56.057407 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:56.058011 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:56.058240 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:56.061134 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:56:56.061141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:56:56.061157 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:56:56.061194 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:56:56.063128 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:56.063739 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:56.063746 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:56.063752 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:56.063883 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:56.064216 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:56.064220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:56.064222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:56.064704 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:56:56.064977 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 22:56:56.065309 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 22:56:56.065778 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":184,"threads_started":1}
I20260925 22:56:56.068125 778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287: No bootstrap required, opened a new log
I20260925 22:56:56.072337 778 raft_consensus.cc:348] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f833443ce16648f08892528119e66287" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45753 } }
I20260925 22:56:56.072741 778 raft_consensus.cc:374] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:56:56.072832 778 raft_consensus.cc:729] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f833443ce16648f08892528119e66287, State: Initialized, Role: FOLLOWER
I20260925 22:56:56.073410 778 consensus_queue.cc:260] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f833443ce16648f08892528119e66287" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45753 } }
I20260925 22:56:56.073565 778 raft_consensus.cc:388] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:56:56.073657 778 raft_consensus.cc:482] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:56:56.073763 778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:56:56.074847 778 raft_consensus.cc:504] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f833443ce16648f08892528119e66287" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45753 } }
I20260925 22:56:56.075260 778 leader_election.cc:302] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f833443ce16648f08892528119e66287; no voters:
I20260925 22:56:56.075623 778 leader_election.cc:288] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:56:56.075789 783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:56:56.076013 783 raft_consensus.cc:686] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [term 1 LEADER]: Becoming Leader. State: Replica: f833443ce16648f08892528119e66287, State: Running, Role: LEADER
I20260925 22:56:56.076439 783 consensus_queue.cc:237] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f833443ce16648f08892528119e66287" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45753 } }
I20260925 22:56:56.077139 778 sys_catalog.cc:564] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:56:56.077893 783 sys_catalog.cc:455] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f833443ce16648f08892528119e66287. Latest consensus state: current_term: 1 leader_uuid: "f833443ce16648f08892528119e66287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f833443ce16648f08892528119e66287" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45753 } } }
I20260925 22:56:56.078127 783 sys_catalog.cc:458] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [sys.catalog]: This master's current role is: LEADER
I20260925 22:56:56.078420 784 sys_catalog.cc:455] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f833443ce16648f08892528119e66287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f833443ce16648f08892528119e66287" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45753 } } }
I20260925 22:56:56.078605 792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:56:56.078603 784 sys_catalog.cc:458] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [sys.catalog]: This master's current role is: LEADER
I20260925 22:56:56.082559 792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:56:56.082980 778 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 22:56:56.100361 792 catalog_manager.cc:1347] Generated new cluster ID: 48d855cef47640a68fb762577714b296
I20260925 22:56:56.100466 792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:56:56.124956 792 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:56:56.126415 792 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:56:56.136814 792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287: Generated new TSK 0
I20260925 22:56:56.138039 792 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.114.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
I20260925 22:56:56.938767 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.114.212:52321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:45753
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:44771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:56:57.096907 823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:56:57.097373 823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:56:57.097465 823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:56:57.103435 823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:56:57.103530 823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:56:57.103626 823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:56:57.110155 823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:44771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.114.212:52321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:45753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:56:57.111383 823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:56:57.112516 823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:56:57.124943 831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:56:57.124938 828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:56:57.125633 829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:56:57.126091 823 server_base.cc:1034] running on GCE node
I20260925 22:56:57.126688 823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:56:57.127596 823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:56:57.128806 823 hybrid_clock.cc:648] HybridClock initialized: now 1790377017128761 us; error 65 us; skew 500 ppm
Sep 25 22:56:57 dist-test-slave-x1lh krb5kdc[32734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377017, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:56:57.132913 823 init.cc:377] Logged in from keytab as kudu/127.20.114.193@KRBTEST.COM (short username kudu)
I20260925 22:56:57.134194 823 webserver.cc:460] Webserver started at http://127.20.114.193:37469/ using document root <none> and password file <none>
I20260925 22:56:57.134943 823 fs_manager.cc:359] Metadata directory not provided
I20260925 22:56:57.135123 823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:56:57.135422 823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:56:57 dist-test-slave-x1lh krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377017, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for HTTP/127.20.114.212@KRBTEST.COM
I20260925 22:56:57.153934 823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "f705b7b8fe2640079388358830f32b3c"
format_stamp: "Formatted at 2026-09-25 22:56:57 on dist-test-slave-x1lh"
server_key: "bf4c0a650b3cf6dc2e0e64ae3326fcea"
server_key_iv: "79fd0701698082c0b9477f6b6eab2835"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:57.154876 823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "f705b7b8fe2640079388358830f32b3c"
format_stamp: "Formatted at 2026-09-25 22:56:57 on dist-test-slave-x1lh"
server_key: "bf4c0a650b3cf6dc2e0e64ae3326fcea"
server_key_iv: "79fd0701698082c0b9477f6b6eab2835"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:57.160699 823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 22:56:57.183322 840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:57.185528 823 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260925 22:56:57.185732 823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "f705b7b8fe2640079388358830f32b3c"
format_stamp: "Formatted at 2026-09-25 22:56:57 on dist-test-slave-x1lh"
server_key: "bf4c0a650b3cf6dc2e0e64ae3326fcea"
server_key_iv: "79fd0701698082c0b9477f6b6eab2835"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:57.185907 823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:56:57.202718 823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:56:57.206807 823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:56:57.207086 823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:56:57.207928 823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:56:57.209245 823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:56:57.209373 823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:57.209506 823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:56:57.209590 823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:57.221474 823 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:40627
I20260925 22:56:57.222905 823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:56:57.230479 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 823
I20260925 22:56:57.230624 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:56:57.230969 20939 external_mini_cluster.cc:1428] Setting key 9566204f2116dcf604244e84190cd6c0
I20260925 22:56:57.232012 953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:40627 every 8 connection(s)
WARNING: no policy specified for kudu/127.20.114.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.194@KRBTEST.COM" created.
Sep 25 22:56:57 dist-test-slave-x1lh krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377017, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:56:57.254077 958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:57.234886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:45753 (local address 127.20.114.193:37019)
0925 22:56:57.235715 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0925 22:56:57.235792 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0925 22:56:57.237269 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:56:57.238237 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:56:57.238250 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:56:57.238690 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:56:57.239550 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:57.239578 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:57.241024 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:57.241028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:57.242747 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:57.242767 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:57.243539 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:57.243544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:57.243777 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:57.244897 (+ 1120us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:56:57.244925 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:56:57.247602 (+ 2677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:56:57.250175 (+ 2573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:56:57.250183 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:56:57.250206 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:56:57.250651 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:56:57.251032 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:56:57.251036 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:56:57.251038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:56:57.251266 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:56:57.251798 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:56:57.251807 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:56:57.252198 (+ 391us) client_negotiation.cc:769] Sending connection context
0925 22:56:57.252420 (+ 222us) client_negotiation.cc:240] Negotiation successful
0925 22:56:57.252835 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":200,"threads_started":1}
I20260925 22:56:57.255754 956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:57.234272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:37019 (local address 127.20.114.254:45753)
0925 22:56:57.235171 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0925 22:56:57.235176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:56:57.237614 (+ 2438us) server_negotiation.cc:407] Connection header received
0925 22:56:57.237674 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:56:57.237680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:56:57.237752 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:56:57.237842 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:56:57.239787 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:57.240783 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:57.242936 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:57.243311 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:57.245233 (+ 1922us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:57.247888 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:56:57.247896 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:56:57.247903 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:56:57.247945 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:56:57.249991 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:57.250805 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:57.250809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:57.250811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:57.250913 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:57.251395 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:57.251398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:57.251399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:57.251665 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:56:57.255264 (+ 3599us) server_negotiation.cc:1035] Waiting for connection context
0925 22:56:57.255381 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 22:56:57.255571 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":86,"threads_started":1}
I20260925 22:56:57.257354 954 heartbeater.cc:347] Connected to a master server at 127.20.114.254:45753
I20260925 22:56:57.257642 954 heartbeater.cc:464] Registering TS with master...
I20260925 22:56:57.258268 954 heartbeater.cc:511] Master 127.20.114.254:45753 requested a full tablet report, sending...
I20260925 22:56:57.260340 742 ts_manager.cc:194] Registered new tserver with Master: f705b7b8fe2640079388358830f32b3c (127.20.114.193:40627)
I20260925 22:56:57.262058 742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:37019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
I20260925 22:56:57.281214 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.114.212:52321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:45753
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:44771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:56:57 dist-test-slave-x1lh krb5kdc[32734](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790377017, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 22:56:57.440058 962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:56:57.440471 962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:56:57.440549 962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:56:57.445997 962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:56:57.446105 962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:56:57.446202 962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:56:57.454695 962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:44771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.114.212:52321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:45753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:56:57.456060 962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:56:57.457576 962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:56:57.470435 971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:56:57.470858 968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:56:57.471184 969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:56:57.471621 962 server_base.cc:1034] running on GCE node
I20260925 22:56:57.472283 962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:56:57.473218 962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:56:57.474401 962 hybrid_clock.cc:648] HybridClock initialized: now 1790377017474381 us; error 54 us; skew 500 ppm
Sep 25 22:56:57 dist-test-slave-x1lh krb5kdc[32734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377017, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:56:57.478477 962 init.cc:377] Logged in from keytab as kudu/127.20.114.194@KRBTEST.COM (short username kudu)
I20260925 22:56:57.479880 962 webserver.cc:460] Webserver started at http://127.20.114.194:40931/ using document root <none> and password file <none>
I20260925 22:56:57.480712 962 fs_manager.cc:359] Metadata directory not provided
I20260925 22:56:57.480821 962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:56:57.481135 962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:56:57 dist-test-slave-x1lh krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377017, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for HTTP/127.20.114.212@KRBTEST.COM
I20260925 22:56:57.507030 962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "f9902d3209de4e37a0d7f5e5103bcc31"
format_stamp: "Formatted at 2026-09-25 22:56:57 on dist-test-slave-x1lh"
server_key: "593b03e0a8b09898109ac271d7fdc76b"
server_key_iv: "2f1968751f3618fb4a5a416ffbaf8a29"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:57.511507 962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "f9902d3209de4e37a0d7f5e5103bcc31"
format_stamp: "Formatted at 2026-09-25 22:56:57 on dist-test-slave-x1lh"
server_key: "593b03e0a8b09898109ac271d7fdc76b"
server_key_iv: "2f1968751f3618fb4a5a416ffbaf8a29"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:57.518653 962 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260925 22:56:57.538617 978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:57.540655 962 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260925 22:56:57.540956 962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "f9902d3209de4e37a0d7f5e5103bcc31"
format_stamp: "Formatted at 2026-09-25 22:56:57 on dist-test-slave-x1lh"
server_key: "593b03e0a8b09898109ac271d7fdc76b"
server_key_iv: "2f1968751f3618fb4a5a416ffbaf8a29"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:57.541245 962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:56:57.559662 962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:56:57.565891 962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:56:57.566143 962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:56:57.566838 962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:56:57.567880 962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:56:57.567953 962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:57.568048 962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:56:57.568109 962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:57.583138 962 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:46231
I20260925 22:56:57.584613 962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:56:57.586570 1091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:46231 every 8 connection(s)
I20260925 22:56:57.594631 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 962
I20260925 22:56:57.594771 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:56:57.595112 20939 external_mini_cluster.cc:1428] Setting key 731129ca829ab2b23ab0e85bfdd7ed41
Sep 25 22:56:57 dist-test-slave-x1lh krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377017, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:56:57.608498 956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:57.587013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:36495 (local address 127.20.114.254:45753)
0925 22:56:57.588607 (+ 1594us) server_negotiation.cc:206] Beginning negotiation
0925 22:56:57.588613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:56:57.588633 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:56:57.589420 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:56:57.589423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:56:57.589489 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:56:57.589581 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:56:57.590850 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:57.591758 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:57.593808 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:57.594177 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:57.594275 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:57.603233 (+ 8958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:56:57.603241 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:56:57.603244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:56:57.603286 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:56:57.605406 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:57.606313 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:57.606317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:57.606319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:57.606429 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:57.606930 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:57.606934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:57.606936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:57.607187 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:56:57.607275 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 22:56:57.608125 (+ 850us) server_negotiation.cc:299] Negotiation successful
0925 22:56:57.608299 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1483}
I20260925 22:56:57.610786 1094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:57.587520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:45753 (local address 127.20.114.194:36495)
0925 22:56:57.588068 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0925 22:56:57.588126 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:56:57.589171 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:56:57.589785 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:56:57.589793 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:56:57.590132 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:56:57.590674 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:57.590690 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:57.592005 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:57.592009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:57.593624 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:57.593639 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:57.594535 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:57.594539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:57.594797 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:57.595782 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:56:57.595817 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:56:57.602906 (+ 7089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:56:57.605624 (+ 2718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:56:57.605633 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:56:57.605658 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:56:57.606147 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:56:57.606573 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:56:57.606577 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:56:57.606579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:56:57.606802 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:56:57.607351 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:56:57.607359 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:56:57.607780 (+ 421us) client_negotiation.cc:769] Sending connection context
0925 22:56:57.608993 (+ 1213us) client_negotiation.cc:240] Negotiation successful
0925 22:56:57.609440 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":164,"threads_started":1}
I20260925 22:56:57.617020 1092 heartbeater.cc:347] Connected to a master server at 127.20.114.254:45753
I20260925 22:56:57.617447 1092 heartbeater.cc:464] Registering TS with master...
I20260925 22:56:57.618223 1092 heartbeater.cc:511] Master 127.20.114.254:45753 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.114.195@KRBTEST.COM; defaulting to no policy
I20260925 22:56:57.619874 742 ts_manager.cc:194] Registered new tserver with Master: f9902d3209de4e37a0d7f5e5103bcc31 (127.20.114.194:46231)
I20260925 22:56:57.620819 742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:36495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
I20260925 22:56:57.648936 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.114.212:52321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:45753
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:44771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:56:57.828645 1097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:56:57.829209 1097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:56:57.829331 1097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:56:57.836764 1097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:56:57.836971 1097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:56:57.837126 1097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:56:57.846952 1097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:44771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.114.212:52321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:45753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:56:57.848770 1097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:56:57.850471 1097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:56:57.867396 1103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:56:57.867398 1105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:56:57.868300 1102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:56:57.870699 1097 server_base.cc:1034] running on GCE node
I20260925 22:56:57.871330 1097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:56:57.872207 1097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:56:57.887723 1097 hybrid_clock.cc:648] HybridClock initialized: now 1790377017887577 us; error 143 us; skew 500 ppm
Sep 25 22:56:57 dist-test-slave-x1lh krb5kdc[32734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377017, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:56:57.891832 1097 init.cc:377] Logged in from keytab as kudu/127.20.114.195@KRBTEST.COM (short username kudu)
I20260925 22:56:57.893113 1097 webserver.cc:460] Webserver started at http://127.20.114.195:33957/ using document root <none> and password file <none>
I20260925 22:56:57.894065 1097 fs_manager.cc:359] Metadata directory not provided
I20260925 22:56:57.894169 1097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:56:57.894452 1097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:56:57 dist-test-slave-x1lh krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377017, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for HTTP/127.20.114.212@KRBTEST.COM
Sep 25 22:56:57 dist-test-slave-x1lh krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377017, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for HTTP/127.20.114.212@KRBTEST.COM
I20260925 22:56:57.921452 1097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "1c9b565aac4d4c0781f9c7697485c069"
format_stamp: "Formatted at 2026-09-25 22:56:57 on dist-test-slave-x1lh"
server_key: "e5cb8e506d0f39770499046b3107b60b"
server_key_iv: "952de36739729412d7a080ee0d8d5e5d"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:57.922305 1097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "1c9b565aac4d4c0781f9c7697485c069"
format_stamp: "Formatted at 2026-09-25 22:56:57 on dist-test-slave-x1lh"
server_key: "e5cb8e506d0f39770499046b3107b60b"
server_key_iv: "952de36739729412d7a080ee0d8d5e5d"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:57.942330 1097 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.005s sys 0.000s
I20260925 22:56:57.976642 1112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:57.981324 1097 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.003s sys 0.001s
I20260925 22:56:57.981606 1097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "1c9b565aac4d4c0781f9c7697485c069"
format_stamp: "Formatted at 2026-09-25 22:56:57 on dist-test-slave-x1lh"
server_key: "e5cb8e506d0f39770499046b3107b60b"
server_key_iv: "952de36739729412d7a080ee0d8d5e5d"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:57.981973 1097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:56:58.021245 1097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:56:58.026288 1097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:56:58.026593 1097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:56:58.027405 1097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:56:58.028644 1097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:56:58.028755 1097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:58.028882 1097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:56:58.028951 1097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:58.041153 1097 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:45533
I20260925 22:56:58.042511 1097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:56:58.051386 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 1097
I20260925 22:56:58.051523 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:56:58.051834 20939 external_mini_cluster.cc:1428] Setting key cfe1a47a4725135d2eb32e411b2d9c21
I20260925 22:56:58.056475 1225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:45533 every 8 connection(s)
Sep 25 22:56:58 dist-test-slave-x1lh krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377017, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:56:58.080600 956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:58.060573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:42177 (local address 127.20.114.254:45753)
0925 22:56:58.060720 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:56:58.060725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:56:58.061338 (+ 613us) server_negotiation.cc:407] Connection header received
0925 22:56:58.063497 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:56:58.063502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:56:58.063570 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:56:58.063628 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:56:58.067572 (+ 3944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:58.068444 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:58.070558 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:58.070899 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:58.070982 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:58.074374 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:56:58.074379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:56:58.074381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:56:58.074413 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:56:58.076230 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:58.077014 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:58.077018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:58.077020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:58.077124 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:58.078832 (+ 1708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:58.078836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:58.078838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:58.079119 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:56:58.079203 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 22:56:58.080263 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0925 22:56:58.080430 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:56:58.081800 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:58.059232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:45753 (local address 127.20.114.195:42177)
0925 22:56:58.061141 (+ 1909us) negotiation.cc:107] Waiting for socket to connect
0925 22:56:58.061215 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0925 22:56:58.063172 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:56:58.064135 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:56:58.064146 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:56:58.064564 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:56:58.067368 (+ 2804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:58.067403 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:58.068770 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:58.068774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:58.070362 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:58.070381 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:58.071269 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:58.071273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:58.071499 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:58.072288 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:56:58.072311 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:56:58.074127 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:56:58.076407 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:56:58.076414 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:56:58.076435 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:56:58.076826 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:56:58.078307 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:56:58.078311 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:56:58.078313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:56:58.078715 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:56:58.079518 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:56:58.079526 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:56:58.079918 (+ 392us) client_negotiation.cc:769] Sending connection context
0925 22:56:58.080147 (+ 229us) client_negotiation.cc:240] Negotiation successful
0925 22:56:58.080573 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1647,"thread_start_us":180,"threads_started":1}
I20260925 22:56:58.083086 1226 heartbeater.cc:347] Connected to a master server at 127.20.114.254:45753
I20260925 22:56:58.083398 1226 heartbeater.cc:464] Registering TS with master...
I20260925 22:56:58.084146 1226 heartbeater.cc:511] Master 127.20.114.254:45753 requested a full tablet report, sending...
I20260925 22:56:58.085471 742 ts_manager.cc:194] Registered new tserver with Master: 1c9b565aac4d4c0781f9c7697485c069 (127.20.114.195:45533)
I20260925 22:56:58.086215 742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:42177
I20260925 22:56:58.092123 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:56:58.093068 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 823
I20260925 22:56:58.100248 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 962
I20260925 22:56:58.109381 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 1097
I20260925 22:56:58.116789 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 710
I20260925 22:56:58.125739 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:45753
--webserver_interface=127.20.114.254
--webserver_port=45539
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:44771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:45753
--ranger_config_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.114.212:52321/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.114.212:52321/kms with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:56:58.249020 1229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:56:58.249384 1229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:56:58.249490 1229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:56:58.254019 1229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:56:58.254139 1229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:56:58.254180 1229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:56:58.254242 1229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:56:58.254300 1229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:56:58.260612 1229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:44771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.114.212:52321/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:45753
--ranger_config_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:45753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=45539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:56:58.261982 1229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:56:58.263172 1229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:56:58.274735 1234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:56:58.274654 1237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:56:58.274654 1235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:56:58.275648 1229 server_base.cc:1034] running on GCE node
I20260925 22:56:58.276325 1229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:56:58.277697 1229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:56:58.278905 1229 hybrid_clock.cc:648] HybridClock initialized: now 1790377018278887 us; error 85 us; skew 500 ppm
Sep 25 22:56:58 dist-test-slave-x1lh krb5kdc[32734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377018, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:56:58.282958 1229 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:56:58.284140 1229 webserver.cc:460] Webserver started at http://127.20.114.254:45539/ using document root <none> and password file <none>
I20260925 22:56:58.285189 1229 fs_manager.cc:359] Metadata directory not provided
I20260925 22:56:58.285305 1229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:56:58 dist-test-slave-x1lh krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377018, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for HTTP/127.20.114.212@KRBTEST.COM
I20260925 22:56:58.340287 1229 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:56:58.343488 1245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:58.345403 1229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:56:58.345569 1229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "f833443ce16648f08892528119e66287"
format_stamp: "Formatted at 2026-09-25 22:56:55 on dist-test-slave-x1lh"
server_key: "e568926063aeeb5968b3b0868367392b"
server_key_iv: "193937e34af6102029b7ccf16351648e"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:58.346365 1229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:56:58.366667 1229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:56:58.369787 1229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:56:58.369972 1229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:56:58.377830 1297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:45753 every 8 connection(s)
I20260925 22:56:58.377827 1229 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:45753
I20260925 22:56:58.379258 1229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:56:58.384433 1298 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:56:58.384516 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 1229
I20260925 22:56:58.385051 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.114.212:52321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:40627
--local_ip_for_outbound_sockets=127.20.114.193
--tserver_master_addrs=127.20.114.254:45753
--webserver_port=37469
--webserver_interface=127.20.114.193
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:44771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.114.212:52321/kms with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:56:58.387086 1298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287: Bootstrap starting.
I20260925 22:56:58.404690 1298 log.cc:824] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287: Log is configured to *not* fsync() on all Append() calls
I20260925 22:56:58.412230 1298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:56:58.412707 1298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287: Bootstrap complete.
I20260925 22:56:58.417055 1298 raft_consensus.cc:348] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f833443ce16648f08892528119e66287" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45753 } }
I20260925 22:56:58.417490 1298 raft_consensus.cc:729] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f833443ce16648f08892528119e66287, State: Initialized, Role: FOLLOWER
I20260925 22:56:58.418073 1298 consensus_queue.cc:260] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f833443ce16648f08892528119e66287" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45753 } }
I20260925 22:56:58.418200 1298 raft_consensus.cc:388] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:56:58.418288 1298 raft_consensus.cc:482] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:56:58.418383 1298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:56:58.420619 1298 raft_consensus.cc:504] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f833443ce16648f08892528119e66287" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45753 } }
I20260925 22:56:58.421039 1298 leader_election.cc:302] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f833443ce16648f08892528119e66287; no voters:
I20260925 22:56:58.421414 1298 leader_election.cc:288] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:56:58.421517 1302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:56:58.421803 1302 raft_consensus.cc:686] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [term 2 LEADER]: Becoming Leader. State: Replica: f833443ce16648f08892528119e66287, State: Running, Role: LEADER
I20260925 22:56:58.422195 1302 consensus_queue.cc:237] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f833443ce16648f08892528119e66287" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45753 } }
I20260925 22:56:58.422905 1298 sys_catalog.cc:564] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:56:58.423664 1304 sys_catalog.cc:455] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f833443ce16648f08892528119e66287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f833443ce16648f08892528119e66287" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45753 } } }
I20260925 22:56:58.423820 1304 sys_catalog.cc:458] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [sys.catalog]: This master's current role is: LEADER
I20260925 22:56:58.424105 1303 sys_catalog.cc:455] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f833443ce16648f08892528119e66287. Latest consensus state: current_term: 2 leader_uuid: "f833443ce16648f08892528119e66287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f833443ce16648f08892528119e66287" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45753 } } }
I20260925 22:56:58.424186 1303 sys_catalog.cc:458] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287 [sys.catalog]: This master's current role is: LEADER
I20260925 22:56:58.424606 1309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:56:58.428184 1309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:56:58.428804 1298 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 22:56:58.428944 1309 catalog_manager.cc:1259] Loaded cluster ID: 48d855cef47640a68fb762577714b296
I20260925 22:56:58.428997 1309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:56:58.431586 1309 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:56:58.432340 1309 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287: Loaded TSK: 0
I20260925 22:56:58.436182 1309 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:56:58.538668 1300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:56:58.538980 1300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:56:58.539093 1300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:56:58.544217 1300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:56:58.544319 1300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:56:58.544425 1300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:56:58.550438 1300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:44771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.114.212:52321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:40627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=37469
--tserver_master_addrs=127.20.114.254:45753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:56:58.551978 1300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:56:58.553896 1300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:56:58.566749 1335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:56:58.567201 1300 server_base.cc:1034] running on GCE node
W20260925 22:56:58.566749 1336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:56:58.567483 1338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:56:58.568007 1300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:56:58.568914 1300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:56:58.570150 1300 hybrid_clock.cc:648] HybridClock initialized: now 1790377018570111 us; error 55 us; skew 500 ppm
Sep 25 22:56:58 dist-test-slave-x1lh krb5kdc[32734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377018, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:56:58.573537 1300 init.cc:377] Logged in from keytab as kudu/127.20.114.193@KRBTEST.COM (short username kudu)
I20260925 22:56:58.574702 1300 webserver.cc:460] Webserver started at http://127.20.114.193:37469/ using document root <none> and password file <none>
I20260925 22:56:58.575551 1300 fs_manager.cc:359] Metadata directory not provided
I20260925 22:56:58.575752 1300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:56:58 dist-test-slave-x1lh krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377018, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for HTTP/127.20.114.212@KRBTEST.COM
I20260925 22:56:58.605234 1300 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:56:58.611462 1346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:58.613709 1300 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 22:56:58.613926 1300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "f705b7b8fe2640079388358830f32b3c"
format_stamp: "Formatted at 2026-09-25 22:56:57 on dist-test-slave-x1lh"
server_key: "bf4c0a650b3cf6dc2e0e64ae3326fcea"
server_key_iv: "79fd0701698082c0b9477f6b6eab2835"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:58.614943 1300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:56:58.636415 1300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:56:58.640463 1300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:56:58.640794 1300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:56:58.641701 1300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:56:58.642931 1300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:56:58.643028 1300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:58.643113 1300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:56:58.643163 1300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:58.655061 1300 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:40627
I20260925 22:56:58.656661 1300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:56:58.664599 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 1300
I20260925 22:56:58.665257 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.114.212:52321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:46231
--local_ip_for_outbound_sockets=127.20.114.194
--tserver_master_addrs=127.20.114.254:45753
--webserver_port=40931
--webserver_interface=127.20.114.194
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:44771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.114.212:52321/kms with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:56:58.668155 1459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:40627 every 8 connection(s)
Sep 25 22:56:58 dist-test-slave-x1lh krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377018, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:56:58.700510 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:58.671497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:45753 (local address 127.20.114.193:47437)
0925 22:56:58.676997 (+ 5500us) negotiation.cc:107] Waiting for socket to connect
0925 22:56:58.677116 (+ 119us) client_negotiation.cc:174] Beginning negotiation
0925 22:56:58.678556 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:56:58.681104 (+ 2548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:56:58.681116 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:56:58.681532 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:56:58.682287 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:58.682312 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:58.684415 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:58.684419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:58.686065 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:58.686082 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:58.687620 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:58.687624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:58.687856 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:58.688693 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:56:58.688718 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:56:58.693289 (+ 4571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:56:58.696440 (+ 3151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:56:58.696448 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:56:58.696479 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:56:58.696931 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:56:58.697377 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:56:58.697380 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:56:58.697382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:56:58.697581 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:56:58.698440 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:56:58.698449 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:56:58.698802 (+ 353us) client_negotiation.cc:769] Sending connection context
0925 22:56:58.699009 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 22:56:58.699377 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5123,"thread_start_us":205,"threads_started":1}
I20260925 22:56:58.702848 1464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:58.673704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:47437 (local address 127.20.114.254:45753)
0925 22:56:58.674409 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0925 22:56:58.674435 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 22:56:58.677548 (+ 3113us) server_negotiation.cc:407] Connection header received
0925 22:56:58.679346 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:56:58.679364 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:56:58.679934 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:56:58.680374 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:56:58.682545 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:58.684200 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:58.686265 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:58.687135 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:58.687331 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:58.693679 (+ 6348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:56:58.693688 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:56:58.693712 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:56:58.693759 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:56:58.696234 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:58.697094 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:58.697101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:58.697108 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:58.697272 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:58.697690 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:58.697693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:58.697695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:58.698232 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:56:58.698371 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 22:56:58.701233 (+ 2862us) server_negotiation.cc:299] Negotiation successful
0925 22:56:58.701607 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":161,"threads_started":1}
I20260925 22:56:58.710744 1460 heartbeater.cc:347] Connected to a master server at 127.20.114.254:45753
I20260925 22:56:58.711037 1460 heartbeater.cc:464] Registering TS with master...
I20260925 22:56:58.711673 1460 heartbeater.cc:511] Master 127.20.114.254:45753 requested a full tablet report, sending...
W20260925 22:56:58.714370 1460 heartbeater.cc:644] Failed to heartbeat to 127.20.114.254:45753 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 22:56:58.714524 1460 heartbeater.cc:464] Registering TS with master...
I20260925 22:56:58.714730 1460 heartbeater.cc:511] Master 127.20.114.254:45753 requested a full tablet report, sending...
I20260925 22:56:58.715873 1460 heartbeater.cc:464] Registering TS with master...
I20260925 22:56:58.716069 1460 heartbeater.cc:511] Master 127.20.114.254:45753 requested a full tablet report, sending...
W20260925 22:56:58.716992 1460 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:56:58.825517 1462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:56:58.825863 1462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:56:58.825944 1462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:56:58.830788 1462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:56:58.830879 1462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:56:58.830982 1462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:56:58.837688 1462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:44771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.114.212:52321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:46231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=40931
--tserver_master_addrs=127.20.114.254:45753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:56:58.839129 1462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:56:58.840490 1462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:56:58.851825 1470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:56:58.851918 1469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:56:58.851825 1472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:56:58.852368 1462 server_base.cc:1034] running on GCE node
I20260925 22:56:58.852880 1462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:56:58.853750 1462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:56:58.854960 1462 hybrid_clock.cc:648] HybridClock initialized: now 1790377018854916 us; error 65 us; skew 500 ppm
Sep 25 22:56:58 dist-test-slave-x1lh krb5kdc[32734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377018, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:56:58.858242 1462 init.cc:377] Logged in from keytab as kudu/127.20.114.194@KRBTEST.COM (short username kudu)
I20260925 22:56:58.859402 1462 webserver.cc:460] Webserver started at http://127.20.114.194:40931/ using document root <none> and password file <none>
I20260925 22:56:58.860052 1462 fs_manager.cc:359] Metadata directory not provided
I20260925 22:56:58.860170 1462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:56:58 dist-test-slave-x1lh krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377018, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for HTTP/127.20.114.212@KRBTEST.COM
I20260925 22:56:58.880043 1462 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:56:58.885864 1480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:58.893138 1462 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.000s
I20260925 22:56:58.893349 1462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "f9902d3209de4e37a0d7f5e5103bcc31"
format_stamp: "Formatted at 2026-09-25 22:56:57 on dist-test-slave-x1lh"
server_key: "593b03e0a8b09898109ac271d7fdc76b"
server_key_iv: "2f1968751f3618fb4a5a416ffbaf8a29"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:58.894133 1462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:56:58.952170 1462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:56:58.956540 1462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:56:58.956940 1462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:56:58.957829 1462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:56:58.958963 1462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:56:58.959028 1462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:58.959085 1462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:56:58.959158 1462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260925 22:56:58.972004 1462 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:46231
I20260925 22:56:58.973598 1462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:56:58.975423 1594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:46231 every 8 connection(s)
I20260925 22:56:58.982563 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 1462
I20260925 22:56:58.983129 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.114.212:52321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:45533
--local_ip_for_outbound_sockets=127.20.114.195
--tserver_master_addrs=127.20.114.254:45753
--webserver_port=33957
--webserver_interface=127.20.114.195
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:44771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.114.212:52321/kms with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:56:58 dist-test-slave-x1lh krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377018, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:56:58.998961 1464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:58.979420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:48859 (local address 127.20.114.254:45753)
0925 22:56:58.979558 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:56:58.979563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:56:58.981368 (+ 1805us) server_negotiation.cc:407] Connection header received
0925 22:56:58.985439 (+ 4071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:56:58.985444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:56:58.985522 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:56:58.985577 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:56:58.987575 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:58.988440 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:58.990422 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:58.990808 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:58.990895 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:58.994470 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:56:58.994475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:56:58.994478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:56:58.994514 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:56:58.996326 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:58.997146 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:58.997150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:58.997152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:58.997251 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:58.997655 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:58.997658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:58.997660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:58.997920 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:56:58.998034 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 22:56:58.998633 (+ 599us) server_negotiation.cc:299] Negotiation successful
0925 22:56:58.998787 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:56:59.000089 1597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:58.978087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:45753 (local address 127.20.114.194:48859)
0925 22:56:58.981132 (+ 3045us) negotiation.cc:107] Waiting for socket to connect
0925 22:56:58.981203 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 22:56:58.985064 (+ 3861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:56:58.986106 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:56:58.986119 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:56:58.986531 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:56:58.987335 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:58.987358 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:58.988641 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:58.988644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:58.990230 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:58.990245 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:58.991148 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:58.991152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:58.991393 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:58.992203 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:56:58.992229 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:56:58.994204 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:56:58.996510 (+ 2306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:56:58.996518 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:56:58.996540 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:56:58.997004 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:56:58.997352 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:56:58.997356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:56:58.997358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:56:58.997538 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:56:58.998034 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:56:58.998043 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:56:58.998345 (+ 302us) client_negotiation.cc:769] Sending connection context
0925 22:56:58.998580 (+ 235us) client_negotiation.cc:240] Negotiation successful
0925 22:56:58.998959 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2579,"thread_start_us":178,"threads_started":1}
I20260925 22:56:59.001405 1595 heartbeater.cc:347] Connected to a master server at 127.20.114.254:45753
I20260925 22:56:59.001681 1595 heartbeater.cc:464] Registering TS with master...
I20260925 22:56:59.002238 1595 heartbeater.cc:511] Master 127.20.114.254:45753 requested a full tablet report, sending...
W20260925 22:56:59.003914 1595 heartbeater.cc:644] Failed to heartbeat to 127.20.114.254:45753 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 22:56:59.004065 1595 heartbeater.cc:464] Registering TS with master...
I20260925 22:56:59.004263 1595 heartbeater.cc:511] Master 127.20.114.254:45753 requested a full tablet report, sending...
I20260925 22:56:59.005288 1595 heartbeater.cc:464] Registering TS with master...
I20260925 22:56:59.005491 1595 heartbeater.cc:511] Master 127.20.114.254:45753 requested a full tablet report, sending...
W20260925 22:56:59.006436 1595 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:56:59.143009 1598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:56:59.143561 1598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:56:59.143687 1598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:56:59.150369 1598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:56:59.150570 1598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:56:59.150719 1598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:56:59.159453 1598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:44771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.114.212:52321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:45533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=33957
--tserver_master_addrs=127.20.114.254:45753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:56:59.160989 1598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:56:59.162456 1598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:56:59.175361 1603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:56:59.175382 1604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:56:59.175869 1598 server_base.cc:1034] running on GCE node
W20260925 22:56:59.177452 1606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:56:59.178099 1598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:56:59.178975 1598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:56:59.180189 1598 hybrid_clock.cc:648] HybridClock initialized: now 1790377019180163 us; error 47 us; skew 500 ppm
Sep 25 22:56:59 dist-test-slave-x1lh krb5kdc[32734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377019, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:56:59.184346 1598 init.cc:377] Logged in from keytab as kudu/127.20.114.195@KRBTEST.COM (short username kudu)
I20260925 22:56:59.185750 1598 webserver.cc:460] Webserver started at http://127.20.114.195:33957/ using document root <none> and password file <none>
I20260925 22:56:59.186568 1598 fs_manager.cc:359] Metadata directory not provided
I20260925 22:56:59.186691 1598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:56:59 dist-test-slave-x1lh krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377019, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for HTTP/127.20.114.212@KRBTEST.COM
I20260925 22:56:59.209805 1598 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:56:59.221944 1614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:59.223904 1598 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.000s
I20260925 22:56:59.224083 1598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "1c9b565aac4d4c0781f9c7697485c069"
format_stamp: "Formatted at 2026-09-25 22:56:57 on dist-test-slave-x1lh"
server_key: "e5cb8e506d0f39770499046b3107b60b"
server_key_iv: "952de36739729412d7a080ee0d8d5e5d"
server_key_version: "kuduclusterkey@0"
I20260925 22:56:59.224795 1598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:56:59.259362 1598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:56:59.263175 1598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:56:59.263407 1598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:56:59.264227 1598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:56:59.265261 1598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:56:59.265338 1598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:59.265411 1598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:56:59.265444 1598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:56:59.277307 1598 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:45533
I20260925 22:56:59.277324 1727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:45533 every 8 connection(s)
I20260925 22:56:59.278666 1598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:56:59.287158 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 1598
Sep 25 22:56:59 dist-test-slave-x1lh krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377019, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:56:59.297546 1464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:59.281271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:33525 (local address 127.20.114.254:45753)
0925 22:56:59.281400 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:56:59.281405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:56:59.282550 (+ 1145us) server_negotiation.cc:407] Connection header received
0925 22:56:59.283879 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:56:59.283884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:56:59.283952 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:56:59.284044 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:56:59.285560 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:59.286151 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:59.288029 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:59.288423 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:59.288520 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:59.291904 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:56:59.291907 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:56:59.291909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:56:59.291938 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:56:59.294056 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:59.294852 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:59.294856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:59.294858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:59.294975 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:59.295440 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:59.295446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:59.295449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:59.295746 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:56:59.295861 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 22:56:59.297248 (+ 1387us) server_negotiation.cc:299] Negotiation successful
0925 22:56:59.297398 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:56:59.298892 1737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:59.281827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:45753 (local address 127.20.114.195:33525)
0925 22:56:59.282373 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0925 22:56:59.282432 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:56:59.283590 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:56:59.284262 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:56:59.284271 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:56:59.284613 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:56:59.285308 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:59.285325 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:59.286311 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:59.286315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:59.287775 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:56:59.287789 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:56:59.288723 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:56:59.288725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:56:59.289186 (+ 461us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:59.290061 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:56:59.290090 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:56:59.291684 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:56:59.294223 (+ 2539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:56:59.294231 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:56:59.294257 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:56:59.294701 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:56:59.295103 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:56:59.295105 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:56:59.295107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:56:59.295308 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:56:59.296330 (+ 1022us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:56:59.296339 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:56:59.296819 (+ 480us) client_negotiation.cc:769] Sending connection context
0925 22:56:59.297146 (+ 327us) client_negotiation.cc:240] Negotiation successful
0925 22:56:59.297577 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":183,"threads_started":1}
I20260925 22:56:59.300513 1728 heartbeater.cc:347] Connected to a master server at 127.20.114.254:45753
I20260925 22:56:59.300796 1728 heartbeater.cc:464] Registering TS with master...
I20260925 22:56:59.300814 1739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:56:59.288052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.20.114.254:45753)
0925 22:56:59.288395 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0925 22:56:59.288408 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:56:59.289426 (+ 1018us) server_negotiation.cc:407] Connection header received
0925 22:56:59.289568 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:56:59.289572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:56:59.289641 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:56:59.289754 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:56:59.290835 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:59.291985 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:59.293472 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:56:59.293878 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:56:59.294009 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:56:59.295923 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:56:59.295929 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:56:59.295932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:56:59.295960 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:56:59.298367 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:59.298872 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:59.298877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:59.298879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:59.298993 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:56:59.299577 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:56:59.299581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:56:59.299583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:56:59.299841 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:56:59.300080 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 22:56:59.300283 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 22:56:59.300489 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":129,"threads_started":1}
I20260925 22:56:59.301504 1728 heartbeater.cc:511] Master 127.20.114.254:45753 requested a full tablet report, sending...
I20260925 22:56:59.303427 1261 ts_manager.cc:194] Registered new tserver with Master: 1c9b565aac4d4c0781f9c7697485c069 (127.20.114.195:45533)
I20260925 22:56:59.305243 1261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:33525
Sep 25 22:56:59 dist-test-slave-x1lh krb5kdc[32734](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790377019, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:56:59.717232 1460 heartbeater.cc:464] Registering TS with master...
I20260925 22:56:59.717571 1460 heartbeater.cc:511] Master 127.20.114.254:45753 requested a full tablet report, sending...
I20260925 22:56:59.718390 1261 ts_manager.cc:194] Registered new tserver with Master: f705b7b8fe2640079388358830f32b3c (127.20.114.193:40627)
I20260925 22:56:59.719017 1261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:47437
I20260925 22:57:00.006732 1595 heartbeater.cc:464] Registering TS with master...
I20260925 22:57:00.007090 1595 heartbeater.cc:511] Master 127.20.114.254:45753 requested a full tablet report, sending...
I20260925 22:57:00.008236 1261 ts_manager.cc:194] Registered new tserver with Master: f9902d3209de4e37a0d7f5e5103bcc31 (127.20.114.194:46231)
I20260925 22:57:00.009137 1261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:48859
I20260925 22:57:00.013741 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:57:00.028110 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:00.015714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35194 (local address 127.20.114.254:45753)
0925 22:57:00.016053 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:00.016059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:00.016479 (+ 420us) server_negotiation.cc:407] Connection header received
0925 22:57:00.016621 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:00.016627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:00.016693 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:00.016803 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:00.017874 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.018721 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.019983 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.020368 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.020493 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:00.023491 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:00.023496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:00.023499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:00.023539 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:00.025888 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:00.026487 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:00.026491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:00.026492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:00.026600 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:00.026957 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:00.026960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:00.026962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:00.027225 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:00.027334 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:00.027761 (+ 427us) server_negotiation.cc:299] Negotiation successful
0925 22:57:00.027917 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":117,"threads_started":1}
I20260925 22:57:00.033737 1261 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:57:00.039364 1261 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:57:00.061688 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:00.054093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:45533 (local address 127.0.0.1:38600)
0925 22:57:00.054552 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:00.054616 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:00.054755 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:00.055730 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:00.055737 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:00.055764 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:57:00.055975 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:00.055984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.057984 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.057988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:00.059584 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:00.059598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.061032 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.061036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:00.061184 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:00.061242 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 22:57:00.061348 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 22:57:00.061499 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":38,"thread_start_us":122,"threads_started":1}
I20260925 22:57:00.066810 1759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:00.055818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:40627 (local address 127.0.0.1:49788)
0925 22:57:00.058353 (+ 2535us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:00.058370 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:00.058465 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:00.059416 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:00.059420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:00.059438 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:57:00.059615 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:00.059622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.062438 (+ 2816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.062441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:00.063762 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:00.063775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.066313 (+ 2538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.066317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:00.066427 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:00.066453 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:57:00.066528 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 22:57:00.066618 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1374,"mutex_wait_us":41,"thread_start_us":103,"threads_started":1}
I20260925 22:57:00.067449 1758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:00.055976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49788 (local address 127.20.114.193:40627)
0925 22:57:00.058728 (+ 2752us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:00.058734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:00.058760 (+ 26us) server_negotiation.cc:407] Connection header received
0925 22:57:00.058840 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:00.058846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:00.059085 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:00.059244 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:57:00.059765 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.062282 (+ 2517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.063925 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.065923 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.066064 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:00.066244 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:00.067168 (+ 924us) server_negotiation.cc:299] Negotiation successful
0925 22:57:00.067300 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2642,"thread_start_us":164,"threads_started":1}
I20260925 22:57:00.068313 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:00.054888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38600 (local address 127.20.114.195:45533)
0925 22:57:00.055210 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:00.055218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:00.055243 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:57:00.055320 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:00.055325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:00.055551 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:00.056430 (+ 879us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:57:00.056605 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.057832 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.059760 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.060803 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.067783 (+ 6980us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:00.067966 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:00.068035 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 22:57:00.068158 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":100,"threads_started":1}
I20260925 22:57:00.069334 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:00.056475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:46231 (local address 127.0.0.1:36190)
0925 22:57:00.062354 (+ 5879us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:00.062369 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:00.062453 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:00.063417 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:00.063420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:00.063441 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:57:00.063531 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:00.063538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.065050 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.065054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:00.066161 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:00.066174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.068837 (+ 2663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.068842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:00.068985 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:00.069022 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 22:57:00.069084 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 22:57:00.069177 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5813,"thread_start_us":89,"threads_started":1}
I20260925 22:57:00.070369 1760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:00.056850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36190 (local address 127.20.114.194:46231)
0925 22:57:00.058278 (+ 1428us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:00.058286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:00.062660 (+ 4374us) server_negotiation.cc:407] Connection header received
0925 22:57:00.062771 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:00.062777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:00.063051 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:00.063253 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:57:00.063715 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.064907 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.066334 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.068671 (+ 2337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.069861 (+ 1190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:00.070045 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:00.070105 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 22:57:00.070218 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1307,"thread_start_us":62,"threads_started":1}
I20260925 22:57:00.071120 1375 tablet_service.cc:1467] Processing CreateTablet for tablet 4f4e8197433f41c481eedf08e1f4e552 (DEFAULT_TABLE table=test-table [id=6a131e256ce44e51acf69b5cc8c6aa79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:57:00.073777 1375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f4e8197433f41c481eedf08e1f4e552. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:57:00.074018 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 4f4e8197433f41c481eedf08e1f4e552 (DEFAULT_TABLE table=test-table [id=6a131e256ce44e51acf69b5cc8c6aa79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:57:00.075325 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f4e8197433f41c481eedf08e1f4e552. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:57:00.088229 1761 tablet_bootstrap.cc:492] T 4f4e8197433f41c481eedf08e1f4e552 P f705b7b8fe2640079388358830f32b3c: Bootstrap starting.
I20260925 22:57:00.091050 1761 tablet_bootstrap.cc:654] T 4f4e8197433f41c481eedf08e1f4e552 P f705b7b8fe2640079388358830f32b3c: Neither blocks nor log segments found. Creating new log.
I20260925 22:57:00.092154 1761 log.cc:824] T 4f4e8197433f41c481eedf08e1f4e552 P f705b7b8fe2640079388358830f32b3c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:57:00.093092 1528 tablet_service.cc:1467] Processing CreateTablet for tablet 4f4e8197433f41c481eedf08e1f4e552 (DEFAULT_TABLE table=test-table [id=6a131e256ce44e51acf69b5cc8c6aa79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:57:00.094383 1528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f4e8197433f41c481eedf08e1f4e552. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 22:57:00 dist-test-slave-x1lh krb5kdc[32734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377019, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for HTTP/127.20.114.212@KRBTEST.COM
I20260925 22:57:00.106720 1763 tablet_bootstrap.cc:492] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069: Bootstrap starting.
I20260925 22:57:00.108788 1761 tablet_bootstrap.cc:492] T 4f4e8197433f41c481eedf08e1f4e552 P f705b7b8fe2640079388358830f32b3c: No bootstrap required, opened a new log
I20260925 22:57:00.109174 1761 ts_tablet_manager.cc:1397] T 4f4e8197433f41c481eedf08e1f4e552 P f705b7b8fe2640079388358830f32b3c: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.000s
I20260925 22:57:00.109586 1763 tablet_bootstrap.cc:654] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069: Neither blocks nor log segments found. Creating new log.
I20260925 22:57:00.110636 1763 log.cc:824] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069: Log is configured to *not* fsync() on all Append() calls
I20260925 22:57:00.112018 1764 tablet_bootstrap.cc:492] T 4f4e8197433f41c481eedf08e1f4e552 P f9902d3209de4e37a0d7f5e5103bcc31: Bootstrap starting.
I20260925 22:57:00.115229 1763 tablet_bootstrap.cc:492] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069: No bootstrap required, opened a new log
I20260925 22:57:00.115474 1763 ts_tablet_manager.cc:1397] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 22:57:00.115672 1761 raft_consensus.cc:348] T 4f4e8197433f41c481eedf08e1f4e552 P f705b7b8fe2640079388358830f32b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9b565aac4d4c0781f9c7697485c069" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45533 } } peers { permanent_uuid: "f705b7b8fe2640079388358830f32b3c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 40627 } } peers { permanent_uuid: "f9902d3209de4e37a0d7f5e5103bcc31" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 46231 } }
I20260925 22:57:00.116125 1761 raft_consensus.cc:374] T 4f4e8197433f41c481eedf08e1f4e552 P f705b7b8fe2640079388358830f32b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:57:00.116232 1761 raft_consensus.cc:729] T 4f4e8197433f41c481eedf08e1f4e552 P f705b7b8fe2640079388358830f32b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f705b7b8fe2640079388358830f32b3c, State: Initialized, Role: FOLLOWER
I20260925 22:57:00.116823 1761 consensus_queue.cc:260] T 4f4e8197433f41c481eedf08e1f4e552 P f705b7b8fe2640079388358830f32b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9b565aac4d4c0781f9c7697485c069" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45533 } } peers { permanent_uuid: "f705b7b8fe2640079388358830f32b3c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 40627 } } peers { permanent_uuid: "f9902d3209de4e37a0d7f5e5103bcc31" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 46231 } }
I20260925 22:57:00.118341 1761 ts_tablet_manager.cc:1428] T 4f4e8197433f41c481eedf08e1f4e552 P f705b7b8fe2640079388358830f32b3c: Time spent starting tablet: real 0.009s user 0.001s sys 0.004s
I20260925 22:57:00.118779 1460 heartbeater.cc:503] Master 127.20.114.254:45753 was elected leader, sending a full tablet report...
I20260925 22:57:00.126192 1763 raft_consensus.cc:348] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9b565aac4d4c0781f9c7697485c069" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45533 } } peers { permanent_uuid: "f705b7b8fe2640079388358830f32b3c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 40627 } } peers { permanent_uuid: "f9902d3209de4e37a0d7f5e5103bcc31" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 46231 } }
I20260925 22:57:00.126672 1763 raft_consensus.cc:374] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:57:00.126776 1763 raft_consensus.cc:729] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c9b565aac4d4c0781f9c7697485c069, State: Initialized, Role: FOLLOWER
I20260925 22:57:00.126971 1764 tablet_bootstrap.cc:654] T 4f4e8197433f41c481eedf08e1f4e552 P f9902d3209de4e37a0d7f5e5103bcc31: Neither blocks nor log segments found. Creating new log.
I20260925 22:57:00.127382 1763 consensus_queue.cc:260] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9b565aac4d4c0781f9c7697485c069" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45533 } } peers { permanent_uuid: "f705b7b8fe2640079388358830f32b3c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 40627 } } peers { permanent_uuid: "f9902d3209de4e37a0d7f5e5103bcc31" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 46231 } }
I20260925 22:57:00.127974 1764 log.cc:824] T 4f4e8197433f41c481eedf08e1f4e552 P f9902d3209de4e37a0d7f5e5103bcc31: Log is configured to *not* fsync() on all Append() calls
I20260925 22:57:00.129181 1763 ts_tablet_manager.cc:1428] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260925 22:57:00.129609 1728 heartbeater.cc:503] Master 127.20.114.254:45753 was elected leader, sending a full tablet report...
I20260925 22:57:00.134603 1764 tablet_bootstrap.cc:492] T 4f4e8197433f41c481eedf08e1f4e552 P f9902d3209de4e37a0d7f5e5103bcc31: No bootstrap required, opened a new log
I20260925 22:57:00.134963 1764 ts_tablet_manager.cc:1397] T 4f4e8197433f41c481eedf08e1f4e552 P f9902d3209de4e37a0d7f5e5103bcc31: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.005s
I20260925 22:57:00.140689 1764 raft_consensus.cc:348] T 4f4e8197433f41c481eedf08e1f4e552 P f9902d3209de4e37a0d7f5e5103bcc31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9b565aac4d4c0781f9c7697485c069" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45533 } } peers { permanent_uuid: "f705b7b8fe2640079388358830f32b3c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 40627 } } peers { permanent_uuid: "f9902d3209de4e37a0d7f5e5103bcc31" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 46231 } }
I20260925 22:57:00.141194 1764 raft_consensus.cc:374] T 4f4e8197433f41c481eedf08e1f4e552 P f9902d3209de4e37a0d7f5e5103bcc31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:57:00.141316 1764 raft_consensus.cc:729] T 4f4e8197433f41c481eedf08e1f4e552 P f9902d3209de4e37a0d7f5e5103bcc31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9902d3209de4e37a0d7f5e5103bcc31, State: Initialized, Role: FOLLOWER
I20260925 22:57:00.141985 1764 consensus_queue.cc:260] T 4f4e8197433f41c481eedf08e1f4e552 P f9902d3209de4e37a0d7f5e5103bcc31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9b565aac4d4c0781f9c7697485c069" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45533 } } peers { permanent_uuid: "f705b7b8fe2640079388358830f32b3c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 40627 } } peers { permanent_uuid: "f9902d3209de4e37a0d7f5e5103bcc31" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 46231 } }
I20260925 22:57:00.143355 1764 ts_tablet_manager.cc:1428] T 4f4e8197433f41c481eedf08e1f4e552 P f9902d3209de4e37a0d7f5e5103bcc31: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260925 22:57:00.143935 1595 heartbeater.cc:503] Master 127.20.114.254:45753 was elected leader, sending a full tablet report...
W20260925 22:57:00.171809 1461 tablet.cc:2369] T 4f4e8197433f41c481eedf08e1f4e552 P f705b7b8fe2640079388358830f32b3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:57:00.231755 1596 tablet.cc:2369] T 4f4e8197433f41c481eedf08e1f4e552 P f9902d3209de4e37a0d7f5e5103bcc31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:57:00.241789 1767 raft_consensus.cc:482] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:57:00.242238 1767 raft_consensus.cc:504] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9b565aac4d4c0781f9c7697485c069" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45533 } } peers { permanent_uuid: "f705b7b8fe2640079388358830f32b3c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 40627 } } peers { permanent_uuid: "f9902d3209de4e37a0d7f5e5103bcc31" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 46231 } }
I20260925 22:57:00.243511 1767 leader_election.cc:288] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f705b7b8fe2640079388358830f32b3c (127.20.114.193:40627), f9902d3209de4e37a0d7f5e5103bcc31 (127.20.114.194:46231)
I20260925 22:57:00.250213 1760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:00.244426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:48507 (local address 127.20.114.194:46231)
0925 22:57:00.244734 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:00.244739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:00.245179 (+ 440us) server_negotiation.cc:407] Connection header received
0925 22:57:00.245402 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:00.245405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:00.245460 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:00.245511 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:57:00.245928 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.246720 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.248399 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.249057 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.249274 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:00.249330 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:00.249957 (+ 627us) server_negotiation.cc:299] Negotiation successful
0925 22:57:00.250060 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:57:00.250936 1770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:00.243867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:46231 (local address 127.20.114.195:48507)
0925 22:57:00.245074 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:00.245088 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:00.245254 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:00.245675 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:00.245678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:00.245696 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:57:00.245806 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:00.245812 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.246924 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.246927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:00.248247 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:00.248258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.249662 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.249666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:00.249783 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:00.249810 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:57:00.250648 (+ 838us) client_negotiation.cc:240] Negotiation successful
0925 22:57:00.250751 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1120,"mutex_wait_us":71,"thread_start_us":171,"threads_started":1}
I20260925 22:57:00.251410 1758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:00.244272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:45317 (local address 127.20.114.193:40627)
0925 22:57:00.244395 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:00.244400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:00.244759 (+ 359us) server_negotiation.cc:407] Connection header received
0925 22:57:00.245016 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:00.245020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:00.245075 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:00.245126 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:57:00.245703 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.246568 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.248374 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.249061 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.249177 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:00.249241 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:00.251188 (+ 1947us) server_negotiation.cc:299] Negotiation successful
0925 22:57:00.251289 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:57:00.251924 1771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:00.243813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:40627 (local address 127.20.114.195:45317)
0925 22:57:00.244644 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:00.244660 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:00.244849 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:00.245278 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:00.245280 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:00.245299 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:57:00.245538 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:00.245552 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.246729 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.246733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:00.248212 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:00.248226 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.250888 (+ 2662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.250892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:00.251009 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:00.251033 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:57:00.251665 (+ 632us) client_negotiation.cc:240] Negotiation successful
0925 22:57:00.251755 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":45,"thread_start_us":139,"threads_started":1}
I20260925 22:57:00.251929 1545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f4e8197433f41c481eedf08e1f4e552" candidate_uuid: "1c9b565aac4d4c0781f9c7697485c069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9902d3209de4e37a0d7f5e5103bcc31" is_pre_election: true
I20260925 22:57:00.252341 1545 raft_consensus.cc:2454] T 4f4e8197433f41c481eedf08e1f4e552 P f9902d3209de4e37a0d7f5e5103bcc31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c9b565aac4d4c0781f9c7697485c069 in term 0.
I20260925 22:57:00.252683 1414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f4e8197433f41c481eedf08e1f4e552" candidate_uuid: "1c9b565aac4d4c0781f9c7697485c069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f705b7b8fe2640079388358830f32b3c" is_pre_election: true
I20260925 22:57:00.253110 1615 leader_election.cc:302] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c9b565aac4d4c0781f9c7697485c069, f9902d3209de4e37a0d7f5e5103bcc31; no voters:
I20260925 22:57:00.253158 1414 raft_consensus.cc:2454] T 4f4e8197433f41c481eedf08e1f4e552 P f705b7b8fe2640079388358830f32b3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c9b565aac4d4c0781f9c7697485c069 in term 0.
I20260925 22:57:00.253459 1767 raft_consensus.cc:2781] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:57:00.253604 1767 raft_consensus.cc:482] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:57:00.253670 1767 raft_consensus.cc:3037] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:57:00.255827 1767 raft_consensus.cc:504] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9b565aac4d4c0781f9c7697485c069" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45533 } } peers { permanent_uuid: "f705b7b8fe2640079388358830f32b3c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 40627 } } peers { permanent_uuid: "f9902d3209de4e37a0d7f5e5103bcc31" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 46231 } }
I20260925 22:57:00.256352 1767 leader_election.cc:288] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069 [CANDIDATE]: Term 1 election: Requested vote from peers f705b7b8fe2640079388358830f32b3c (127.20.114.193:40627), f9902d3209de4e37a0d7f5e5103bcc31 (127.20.114.194:46231)
I20260925 22:57:00.257114 1545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f4e8197433f41c481eedf08e1f4e552" candidate_uuid: "1c9b565aac4d4c0781f9c7697485c069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9902d3209de4e37a0d7f5e5103bcc31"
I20260925 22:57:00.257180 1414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f4e8197433f41c481eedf08e1f4e552" candidate_uuid: "1c9b565aac4d4c0781f9c7697485c069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f705b7b8fe2640079388358830f32b3c"
I20260925 22:57:00.257311 1545 raft_consensus.cc:3037] T 4f4e8197433f41c481eedf08e1f4e552 P f9902d3209de4e37a0d7f5e5103bcc31 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:57:00.257359 1414 raft_consensus.cc:3037] T 4f4e8197433f41c481eedf08e1f4e552 P f705b7b8fe2640079388358830f32b3c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:57:00.259102 1545 raft_consensus.cc:2454] T 4f4e8197433f41c481eedf08e1f4e552 P f9902d3209de4e37a0d7f5e5103bcc31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c9b565aac4d4c0781f9c7697485c069 in term 1.
I20260925 22:57:00.259222 1414 raft_consensus.cc:2454] T 4f4e8197433f41c481eedf08e1f4e552 P f705b7b8fe2640079388358830f32b3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c9b565aac4d4c0781f9c7697485c069 in term 1.
I20260925 22:57:00.259642 1615 leader_election.cc:302] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c9b565aac4d4c0781f9c7697485c069, f9902d3209de4e37a0d7f5e5103bcc31; no voters:
I20260925 22:57:00.259884 1767 raft_consensus.cc:2781] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:57:00.260265 1767 raft_consensus.cc:686] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069 [term 1 LEADER]: Becoming Leader. State: Replica: 1c9b565aac4d4c0781f9c7697485c069, State: Running, Role: LEADER
I20260925 22:57:00.260797 1767 consensus_queue.cc:237] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9b565aac4d4c0781f9c7697485c069" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45533 } } peers { permanent_uuid: "f705b7b8fe2640079388358830f32b3c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 40627 } } peers { permanent_uuid: "f9902d3209de4e37a0d7f5e5103bcc31" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 46231 } }
I20260925 22:57:00.267620 1261 catalog_manager.cc:5495] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c9b565aac4d4c0781f9c7697485c069 (127.20.114.195). New cstate: current_term: 1 leader_uuid: "1c9b565aac4d4c0781f9c7697485c069" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9b565aac4d4c0781f9c7697485c069" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f705b7b8fe2640079388358830f32b3c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 40627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9902d3209de4e37a0d7f5e5103bcc31" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 46231 } health_report { overall_health: UNKNOWN } } }
W20260925 22:57:00.280690 1729 tablet.cc:2369] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:57:00.310328 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:00.302891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38602 (local address 127.20.114.195:45533)
0925 22:57:00.303014 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:00.303018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:00.303087 (+ 69us) server_negotiation.cc:407] Connection header received
0925 22:57:00.303300 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:00.303304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:00.303378 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:00.303510 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:57:00.304161 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.304975 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.306351 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:00.306687 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:00.306805 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:00.307216 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:57:00.307856 (+ 640us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:57:00.307994 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:00.308122 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 22:57:00.310159 (+ 2037us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:57:00.318483 1414 raft_consensus.cc:1261] T 4f4e8197433f41c481eedf08e1f4e552 P f705b7b8fe2640079388358830f32b3c [term 1 FOLLOWER]: Refusing update from remote peer 1c9b565aac4d4c0781f9c7697485c069: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:57:00.318483 1545 raft_consensus.cc:1261] T 4f4e8197433f41c481eedf08e1f4e552 P f9902d3209de4e37a0d7f5e5103bcc31 [term 1 FOLLOWER]: Refusing update from remote peer 1c9b565aac4d4c0781f9c7697485c069: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:57:00.319343 1767 consensus_queue.cc:1035] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f705b7b8fe2640079388358830f32b3c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 40627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:57:00.319758 1772 consensus_queue.cc:1035] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9902d3209de4e37a0d7f5e5103bcc31" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:57:00.335515 1776 mvcc.cc:205] Tried to move back new op lower bound from 7333384275215486976 to 7333384274994638848. Current Snapshot: MvccSnapshot[applied={T|T < 7333384275215486976}]
I20260925 22:57:00.338655 1775 mvcc.cc:205] Tried to move back new op lower bound from 7333384275215486976 to 7333384274994638848. Current Snapshot: MvccSnapshot[applied={T|T < 7333384275215486976 or (T in {7333384275215486976})}]
I20260925 22:57:00.348745 1261 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:57:00.348991 1261 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:57:00.352979 1261 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f833443ce16648f08892528119e66287: Sending DeleteTablet for 3 replicas of tablet 4f4e8197433f41c481eedf08e1f4e552
I20260925 22:57:00.354228 1375 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f4e8197433f41c481eedf08e1f4e552 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:57:00 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:49788
I20260925 22:57:00.355422 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f4e8197433f41c481eedf08e1f4e552 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:57:00 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:38600
I20260925 22:57:00.356624 1528 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f4e8197433f41c481eedf08e1f4e552 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:57:00 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:36190
I20260925 22:57:00.358395 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 1300
I20260925 22:57:00.369313 1786 tablet_replica.cc:330] T 4f4e8197433f41c481eedf08e1f4e552 P f9902d3209de4e37a0d7f5e5103bcc31: stopping tablet replica
I20260925 22:57:00.369706 1785 tablet_replica.cc:330] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069: stopping tablet replica
I20260925 22:57:00.370332 1786 raft_consensus.cc:2229] T 4f4e8197433f41c481eedf08e1f4e552 P f9902d3209de4e37a0d7f5e5103bcc31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:57:00.370832 1785 raft_consensus.cc:2229] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:57:00.371297 1786 raft_consensus.cc:2258] T 4f4e8197433f41c481eedf08e1f4e552 P f9902d3209de4e37a0d7f5e5103bcc31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:57:00.371827 1785 raft_consensus.cc:2258] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:57:00.376279 1785 ts_tablet_manager.cc:1905] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:57:00.378213 1786 ts_tablet_manager.cc:1905] T 4f4e8197433f41c481eedf08e1f4e552 P f9902d3209de4e37a0d7f5e5103bcc31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:57:00.382102 1785 ts_tablet_manager.cc:1918] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:57:00.385921 1786 ts_tablet_manager.cc:1918] T 4f4e8197433f41c481eedf08e1f4e552 P f9902d3209de4e37a0d7f5e5103bcc31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260925 22:57:00.387974 1249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.114.193:40627 (error 108)
I20260925 22:57:00.389235 1786 log.cc:1196] T 4f4e8197433f41c481eedf08e1f4e552 P f9902d3209de4e37a0d7f5e5103bcc31: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/4f4e8197433f41c481eedf08e1f4e552
I20260925 22:57:00.389591 1785 log.cc:1196] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/4f4e8197433f41c481eedf08e1f4e552
I20260925 22:57:00.389889 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 1462
I20260925 22:57:00.394259 1785 ts_tablet_manager.cc:1939] T 4f4e8197433f41c481eedf08e1f4e552 P 1c9b565aac4d4c0781f9c7697485c069: Deleting consensus metadata
I20260925 22:57:00.402282 1247 catalog_manager.cc:4841] TS 1c9b565aac4d4c0781f9c7697485c069 (127.20.114.195:45533): tablet 4f4e8197433f41c481eedf08e1f4e552 (table test-table [id=6a131e256ce44e51acf69b5cc8c6aa79]) successfully deleted
I20260925 22:57:00.403566 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:00.403190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:40627 (local address 127.0.0.1:49802)
0925 22:57:00.403352 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:00.403459 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.193:40627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 22:57:00.403779 1249 catalog_manager.cc:4584] TS f705b7b8fe2640079388358830f32b3c (127.20.114.193:40627): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4f4e8197433f41c481eedf08e1f4e552: Network error: Client connection negotiation failed: client connection to 127.20.114.193:40627: connect: Connection refused (error 111)
W20260925 22:57:00.404824 1755 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:57:00.404307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:46231 (local address 127.0.0.1:36200)
0925 22:57:00.404417 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:00.404433 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:00.404540 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:00.404728 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.194:46231: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 22:57:00.405314 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 1598
I20260925 22:57:00.415409 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 1229
I20260925 22:57:00.422583 20939 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 22:57:00.769109 20939 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 22:57:00.769243 20939 mini_ranger.cc:66] Stopping Ranger...
I20260925 22:57:01.115430 20939 mini_ranger.cc:68] Stopped Ranger
2026-09-25 22:57:01.115 UTC [32764] LOG: received smart shutdown request
2026-09-25 22:57:01.119 UTC [32764] LOG: background worker "logical replication launcher" (PID 305) exited with exit code 1
2026-09-25 22:57:01.120 UTC [300] LOG: shutting down
2026-09-25 22:57:01.339 UTC [32764] LOG: database system is shut down
2026-09-25T22:57:01Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53943 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:57:01 dist-test-slave-x1lh krb5kdc[1796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:57:01 dist-test-slave-x1lh krb5kdc[1796](info): set up 2 sockets
Sep 25 22:57:01 dist-test-slave-x1lh krb5kdc[1796](info): commencing operation
krb5kdc: starting...
W20260925 22:57:02.545293 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:57:02 dist-test-slave-x1lh krb5kdc[1796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:57:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:57:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:57:02.685597 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:36469
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:36469,127.20.114.253:33915,127.20.114.252:38453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:57:02.809311 1809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:02.809650 1809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:02.809696 1809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:02.814085 1809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:57:02.814162 1809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:02.814181 1809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:02.814199 1809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:57:02.814214 1809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:57:02.820068 1809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:36469,127.20.114.253:33915,127.20.114.252:38453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:36469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:02.821345 1809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:02.822633 1809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:57:02.832676 1814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:02.833155 1815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:02.833161 1817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:02.833423 1809 server_base.cc:1034] running on GCE node
I20260925 22:57:02.834107 1809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:02.836030 1809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:02.837255 1809 hybrid_clock.cc:648] HybridClock initialized: now 1790377022837218 us; error 58 us; skew 500 ppm
Sep 25 22:57:02 dist-test-slave-x1lh krb5kdc[1796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377022, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:02.841322 1809 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:57:02.842660 1809 webserver.cc:460] Webserver started at http://127.20.114.254:43861/ using document root <none> and password file <none>
I20260925 22:57:02.843431 1809 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:02.843541 1809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:02.843804 1809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:57:02.846195 1809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "0a64b4a9f5f444f7985ff1943f13a52b"
format_stamp: "Formatted at 2026-09-25 22:57:02 on dist-test-slave-x1lh"
server_key: "0da464324c05b8c658615876a8b33116"
server_key_iv: "439da1acec9ca0c5b4f9fc3f77aefbdd"
server_key_version: "encryptionkey@0"
I20260925 22:57:02.846880 1809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "0a64b4a9f5f444f7985ff1943f13a52b"
format_stamp: "Formatted at 2026-09-25 22:57:02 on dist-test-slave-x1lh"
server_key: "0da464324c05b8c658615876a8b33116"
server_key_iv: "439da1acec9ca0c5b4f9fc3f77aefbdd"
server_key_version: "encryptionkey@0"
I20260925 22:57:02.851908 1809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 22:57:02.855412 1824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:02.857406 1809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:57:02.857623 1809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "0a64b4a9f5f444f7985ff1943f13a52b"
format_stamp: "Formatted at 2026-09-25 22:57:02 on dist-test-slave-x1lh"
server_key: "0da464324c05b8c658615876a8b33116"
server_key_iv: "439da1acec9ca0c5b4f9fc3f77aefbdd"
server_key_version: "encryptionkey@0"
I20260925 22:57:02.857797 1809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:02.892447 1809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:02.897113 1809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:02.897356 1809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:02.906476 1809 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:36469
I20260925 22:57:02.906548 1876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:36469 every 8 connection(s)
I20260925 22:57:02.907593 1809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:57:02.911198 1877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:57:02.914004 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 1809
I20260925 22:57:02.914116 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:57:02.914402 20939 external_mini_cluster.cc:1428] Setting key 278e4e18662f92ec724b725c82991b3c
I20260925 22:57:02.915838 1877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36469 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.114.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.253@KRBTEST.COM" created.
Sep 25 22:57:02 dist-test-slave-x1lh krb5kdc[1796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377022, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:57:02.936220 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:02.918919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:36469 (local address 127.0.0.1:54556)
0925 22:57:02.919540 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:02.919601 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:02.920644 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:02.921623 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:02.921628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:02.922010 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:02.922301 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:02.922317 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:02.923947 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:02.923950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:02.925182 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:02.925194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:02.926113 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:02.926116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:02.926219 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:02.927167 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:02.927192 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:02.929271 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:02.931854 (+ 2583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:02.931867 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:02.931880 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:02.932321 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:02.932746 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:02.932749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:02.932751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:02.932955 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:02.934009 (+ 1054us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:02.934016 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:02.934192 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 22:57:02.934388 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 22:57:02.934890 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":136,"thread_start_us":168,"threads_started":1}
I20260925 22:57:02.936220 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:02.919984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54556 (local address 127.20.114.254:36469)
0925 22:57:02.920379 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:02.920390 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:02.920444 (+ 54us) server_negotiation.cc:407] Connection header received
0925 22:57:02.921051 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:02.921058 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:02.921350 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:02.921584 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:02.922495 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:02.923755 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:02.925325 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:02.925817 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:02.925960 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:02.929460 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:02.929466 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:02.929481 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:02.929505 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:02.931645 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:02.932471 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:02.932478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:02.932480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:02.932633 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:02.933340 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:02.933345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:02.933355 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:02.933870 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:02.934035 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:02.934467 (+ 432us) server_negotiation.cc:299] Negotiation successful
0925 22:57:02.934837 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":252,"thread_start_us":104,"threads_started":1}
I20260925 22:57:02.940301 1877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:57:02.941417 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:02.940929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:33915 (local address 127.0.0.1:38862)
0925 22:57:02.941226 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:02.941333 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.253:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
W20260925 22:57:02.941731 1828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.114.253:33915: connect: Connection refused (error 111)
I20260925 22:57:02.942926 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:02.942597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:33915 (local address 127.0.0.1:38864)
0925 22:57:02.942773 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:02.942841 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.253:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 22:57:02.943357 1877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.253:33915: Network error: Client connection negotiation failed: client connection to 127.20.114.253:33915: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.253.keytab.
Entry for principal kudu/127.20.114.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.253.keytab.
I20260925 22:57:02.949451 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.253:33915
--webserver_interface=127.20.114.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.253.keytab
--principal=kudu/127.20.114.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:36469,127.20.114.253:33915,127.20.114.252:38453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:57:02.992692 1877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } attempt: 1
I20260925 22:57:02.993846 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:02.993409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:33915 (local address 127.0.0.1:38874)
0925 22:57:02.993658 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:02.993759 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.253:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260925 22:57:02.994725 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:02.994466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:33915 (local address 127.0.0.1:38876)
0925 22:57:02.994592 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:02.994660 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.253:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 22:57:02.994858 1877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.253:33915: Network error: Client connection negotiation failed: client connection to 127.20.114.253:33915: connect: Connection refused (error 111)
I20260925 22:57:03.063138 1877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } attempt: 2
I20260925 22:57:03.064065 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.063745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:33915 (local address 127.0.0.1:38890)
0925 22:57:03.063899 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.063977 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.253:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 22:57:03.064823 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.064588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:33915 (local address 127.0.0.1:38894)
0925 22:57:03.064704 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.064760 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.253:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 22:57:03.064996 1877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.253:33915: Network error: Client connection negotiation failed: client connection to 127.20.114.253:33915: connect: Connection refused (error 111)
W20260925 22:57:03.076551 1884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:03.076988 1884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:03.077109 1884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:03.081924 1884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:57:03.082079 1884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:03.082124 1884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:03.082165 1884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:57:03.082199 1884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:57:03.088176 1884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:36469,127.20.114.253:33915,127.20.114.252:38453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.253.keytab
--principal=kudu/127.20.114.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.253:33915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.114.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:03.089501 1884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:03.090659 1884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:57:03.100445 1892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:03.100688 1884 server_base.cc:1034] running on GCE node
W20260925 22:57:03.100445 1890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:03.100498 1889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:03.101531 1884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:03.102943 1884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:03.104157 1884 hybrid_clock.cc:648] HybridClock initialized: now 1790377023104137 us; error 45 us; skew 500 ppm
Sep 25 22:57:03 dist-test-slave-x1lh krb5kdc[1796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377023, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:03.107723 1884 init.cc:377] Logged in from keytab as kudu/127.20.114.253@KRBTEST.COM (short username kudu)
I20260925 22:57:03.108996 1884 webserver.cc:460] Webserver started at http://127.20.114.253:41359/ using document root <none> and password file <none>
I20260925 22:57:03.109674 1884 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:03.109782 1884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:03.110056 1884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:57:03.112375 1884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-1/data/instance:
uuid: "7fd052b7bbad4cad9610a68b0cd19e13"
format_stamp: "Formatted at 2026-09-25 22:57:03 on dist-test-slave-x1lh"
server_key: "069cc731d0814814a15859fbbda3c7e1"
server_key_iv: "4a0e5c3763392651af49ec7a8c0433b1"
server_key_version: "encryptionkey@0"
I20260925 22:57:03.113158 1884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-1/wal/instance:
uuid: "7fd052b7bbad4cad9610a68b0cd19e13"
format_stamp: "Formatted at 2026-09-25 22:57:03 on dist-test-slave-x1lh"
server_key: "069cc731d0814814a15859fbbda3c7e1"
server_key_iv: "4a0e5c3763392651af49ec7a8c0433b1"
server_key_version: "encryptionkey@0"
I20260925 22:57:03.117857 1884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:57:03.121271 1899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:03.123165 1884 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:57:03.123386 1884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-1/wal
uuid: "7fd052b7bbad4cad9610a68b0cd19e13"
format_stamp: "Formatted at 2026-09-25 22:57:03 on dist-test-slave-x1lh"
server_key: "069cc731d0814814a15859fbbda3c7e1"
server_key_iv: "4a0e5c3763392651af49ec7a8c0433b1"
server_key_version: "encryptionkey@0"
I20260925 22:57:03.123553 1884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:03.145377 1884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:03.149276 1884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:03.149516 1884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:03.156193 1877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } attempt: 3
I20260925 22:57:03.157229 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.156785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:33915 (local address 127.0.0.1:38904)
0925 22:57:03.157027 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.157118 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.253:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260925 22:57:03.157269 1884 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.253:33915
I20260925 22:57:03.157588 1951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.253:33915 every 8 connection(s)
I20260925 22:57:03.158603 1884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-1/data/info.pb
I20260925 22:57:03.161722 1953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 22:57:03 dist-test-slave-x1lh krb5kdc[1796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377022, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for kudu/127.20.114.253@KRBTEST.COM
I20260925 22:57:03.166368 1953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36469 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:57:03.167481 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 1884
I20260925 22:57:03.167627 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-1/wal/instance
I20260925 22:57:03.167889 20939 external_mini_cluster.cc:1428] Setting key 2cb6ed1bfaab623e8b7273d19789edcb
I20260925 22:57:03.171504 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.157896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:33915 (local address 127.0.0.1:38906)
0925 22:57:03.158034 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.158050 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:03.158154 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:03.160636 (+ 2482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:03.160639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:03.160852 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:03.161040 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:03.161050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.162618 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.162624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:03.163188 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:03.163198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.163983 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.163988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:03.164096 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:03.164687 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:03.164701 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:03.166287 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:03.169206 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:03.169209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:03.169212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:03.169517 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:03.169947 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:03.169953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:03.169957 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:03.170084 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:03.170818 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:03.170822 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:03.170941 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 22:57:03.171066 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 22:57:03.171235 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 22:57:03.172719 1952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.158667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38906 (local address 127.20.114.253:33915)
0925 22:57:03.159189 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:03.159211 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:03.159262 (+ 51us) server_negotiation.cc:407] Connection header received
0925 22:57:03.159928 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:03.159946 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:03.160315 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:03.160704 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:03.161246 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.162475 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.163325 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.163841 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.164033 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:03.166555 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:03.166560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:03.166576 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:03.166605 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:03.169066 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.169657 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.169664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.169670 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.169814 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.170203 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.170208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.170210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.170705 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:03.170850 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:03.171284 (+ 434us) server_negotiation.cc:299] Negotiation successful
0925 22:57:03.171631 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":130,"threads_started":1}
I20260925 22:57:03.175065 1877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:57:03.175989 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.175624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:38453 (local address 127.0.0.1:40484)
0925 22:57:03.175833 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.175902 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
Sep 25 22:57:03 dist-test-slave-x1lh krb5kdc[1796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377023, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.253@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:57:03.176908 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.176625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:38453 (local address 127.0.0.1:40488)
0925 22:57:03.176755 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.176809 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 22:57:03.177031 1877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.252:38453: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.20.114.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.252@KRBTEST.COM" created.
I20260925 22:57:03.180956 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.168502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:36469 (local address 127.0.0.1:54558)
0925 22:57:03.169011 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.169075 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:03.169269 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:03.169714 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:03.169721 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:03.170179 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:03.170367 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:03.170374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.172066 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.172069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:03.173821 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:03.173830 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.174341 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.174345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:03.174485 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:03.175150 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:03.175167 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:03.176825 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:03.178715 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:03.178724 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:03.178730 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:03.179004 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:03.179479 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:03.179482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:03.179484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:03.179586 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:03.180252 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:03.180263 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:03.180385 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 22:57:03.180552 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 22:57:03.180729 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":126,"threads_started":1}
I20260925 22:57:03.181099 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.168582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54558 (local address 127.20.114.254:36469)
0925 22:57:03.168724 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:03.168729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:03.169162 (+ 433us) server_negotiation.cc:407] Connection header received
0925 22:57:03.169418 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:03.169422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:03.169481 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:03.169568 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:03.170499 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.171895 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.173962 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.174215 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.174341 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:03.176979 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:03.176984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:03.176986 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:03.177020 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:03.178586 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.179261 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.179264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.179266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.179352 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.179834 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.179842 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.179844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.180139 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:03.180244 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:03.180678 (+ 434us) server_negotiation.cc:299] Negotiation successful
0925 22:57:03.180827 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:57:03.183166 1953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:57:03 dist-test-slave-x1lh krb5kdc[1796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377023, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.253@KRBTEST.COM for kudu/127.20.114.253@KRBTEST.COM
I20260925 22:57:03.192304 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.183725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:33915 (local address 127.0.0.1:38918)
0925 22:57:03.183847 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.183860 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:03.183943 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:03.184215 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:03.184218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:03.184409 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:03.184485 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:03.184491 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.185296 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.185302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:03.185833 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:03.185841 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.186290 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.186293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:03.186358 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:03.186891 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:03.186913 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:03.188478 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:03.190634 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:03.190638 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:03.190641 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:03.190857 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:03.191167 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:03.191170 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:03.191172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:03.191260 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:03.191684 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:03.191687 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:03.191788 (+ 101us) client_negotiation.cc:769] Sending connection context
0925 22:57:03.191905 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 22:57:03.192095 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 22:57:03.192304 1952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.183804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38918 (local address 127.20.114.253:33915)
0925 22:57:03.184058 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:03.184061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:03.184069 (+ 8us) server_negotiation.cc:407] Connection header received
0925 22:57:03.184091 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:03.184093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:03.184120 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:03.184204 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:03.184595 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.185193 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.185951 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.186177 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.186287 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:03.188662 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:03.188666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:03.188668 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:03.188699 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:03.190483 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.190968 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.190974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.190976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.191060 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.191356 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.191361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.191363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.191574 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:03.191685 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:03.191962 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 22:57:03.192175 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":193}
I20260925 22:57:03.193328 1953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:57:03.194186 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.193870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:38453 (local address 127.0.0.1:40504)
0925 22:57:03.194010 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.194113 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 22:57:03.194793 1900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
I20260925 22:57:03.195803 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.195553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:38453 (local address 127.0.0.1:40520)
0925 22:57:03.195681 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.195741 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.252.keytab.
Entry for principal kudu/127.20.114.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.252.keytab.
W20260925 22:57:03.196035 1953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.252:38453: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
I20260925 22:57:03.196973 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.252:38453
--webserver_interface=127.20.114.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.252.keytab
--principal=kudu/127.20.114.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:36469,127.20.114.253:33915,127.20.114.252:38453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:57:03.208237 1877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } attempt: 1
I20260925 22:57:03.209272 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.208889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:38453 (local address 127.0.0.1:40522)
0925 22:57:03.209059 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.209182 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 22:57:03.210099 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.209850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:38453 (local address 127.0.0.1:40538)
0925 22:57:03.209974 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.210029 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 22:57:03.210260 1877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.252:38453: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
I20260925 22:57:03.245337 1953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } attempt: 1
I20260925 22:57:03.246292 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.245974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:38453 (local address 127.0.0.1:40546)
0925 22:57:03.246143 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.246210 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 22:57:03.247114 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.246862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:38453 (local address 127.0.0.1:40562)
0925 22:57:03.246993 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.247048 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 22:57:03.247247 1953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.252:38453: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
I20260925 22:57:03.285444 1877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } attempt: 2
I20260925 22:57:03.286454 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.286084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:38453 (local address 127.0.0.1:40578)
0925 22:57:03.286294 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.286368 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 22:57:03.287384 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.287119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:38453 (local address 127.0.0.1:40580)
0925 22:57:03.287245 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.287311 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 22:57:03.287544 1877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.252:38453: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
I20260925 22:57:03.315452 1953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } attempt: 2
I20260925 22:57:03.316377 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.316078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:38453 (local address 127.0.0.1:40596)
0925 22:57:03.316214 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.316278 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 22:57:03.317359 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.317091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:38453 (local address 127.0.0.1:40598)
0925 22:57:03.317236 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.317297 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 22:57:03.317533 1953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.252:38453: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
W20260925 22:57:03.321645 1959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:03.321976 1959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:03.322052 1959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:03.327105 1959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:57:03.327203 1959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:03.327225 1959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:03.327244 1959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:57:03.327260 1959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:57:03.334043 1959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:36469,127.20.114.253:33915,127.20.114.252:38453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.252.keytab
--principal=kudu/127.20.114.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.252:38453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.114.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:03.335566 1959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:03.336732 1959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:57:03.347640 1967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:03.347641 1965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:03.347596 1964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:03.347911 1959 server_base.cc:1034] running on GCE node
I20260925 22:57:03.349272 1959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:03.350692 1959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:03.351948 1959 hybrid_clock.cc:648] HybridClock initialized: now 1790377023351906 us; error 62 us; skew 500 ppm
Sep 25 22:57:03 dist-test-slave-x1lh krb5kdc[1796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377023, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:03.355670 1959 init.cc:377] Logged in from keytab as kudu/127.20.114.252@KRBTEST.COM (short username kudu)
I20260925 22:57:03.357018 1959 webserver.cc:460] Webserver started at http://127.20.114.252:40655/ using document root <none> and password file <none>
I20260925 22:57:03.357678 1959 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:03.357782 1959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:03.358053 1959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:57:03.360427 1959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-2/data/instance:
uuid: "234806cba37e458e89748000421701f9"
format_stamp: "Formatted at 2026-09-25 22:57:03 on dist-test-slave-x1lh"
server_key: "7f308327f63489cc5659717da47c00f9"
server_key_iv: "479eba40e43a5d3fc53151be6750a035"
server_key_version: "encryptionkey@0"
I20260925 22:57:03.361284 1959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-2/wal/instance:
uuid: "234806cba37e458e89748000421701f9"
format_stamp: "Formatted at 2026-09-25 22:57:03 on dist-test-slave-x1lh"
server_key: "7f308327f63489cc5659717da47c00f9"
server_key_iv: "479eba40e43a5d3fc53151be6750a035"
server_key_version: "encryptionkey@0"
I20260925 22:57:03.367666 1959 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260925 22:57:03.371886 1974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:03.373910 1959 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 22:57:03.374090 1959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-2/wal
uuid: "234806cba37e458e89748000421701f9"
format_stamp: "Formatted at 2026-09-25 22:57:03 on dist-test-slave-x1lh"
server_key: "7f308327f63489cc5659717da47c00f9"
server_key_iv: "479eba40e43a5d3fc53151be6750a035"
server_key_version: "encryptionkey@0"
I20260925 22:57:03.374282 1959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:03.386723 1877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } attempt: 3
I20260925 22:57:03.387677 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.387323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:38453 (local address 127.0.0.1:40608)
0925 22:57:03.387498 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.387590 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 22:57:03.388756 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.388503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:38453 (local address 127.0.0.1:40618)
0925 22:57:03.388633 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.388686 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 22:57:03.388955 1877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.252:38453: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
I20260925 22:57:03.401408 1959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:03.405385 1959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:03.405579 1959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:03.408725 1953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } attempt: 3
I20260925 22:57:03.409762 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.409344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:38453 (local address 127.0.0.1:40632)
0925 22:57:03.409596 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.409682 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 22:57:03.410710 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.410391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:38453 (local address 127.0.0.1:40636)
0925 22:57:03.410569 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.410627 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260925 22:57:03.410846 1953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.252:38453: Network error: Client connection negotiation failed: client connection to 127.20.114.252:38453: connect: Connection refused (error 111)
I20260925 22:57:03.414162 1959 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.252:38453
I20260925 22:57:03.414494 2026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.252:38453 every 8 connection(s)
I20260925 22:57:03.415315 1959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-2/data/info.pb
I20260925 22:57:03.418658 2027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:57:03.422919 2027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36469 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:57:03.425248 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 1959
I20260925 22:57:03.425386 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376856092611-20939-0/minicluster-data/master-2/wal/instance
I20260925 22:57:03.425673 20939 external_mini_cluster.cc:1428] Setting key 551aa90ddc1ea3e67c735b578e562ad3
Sep 25 22:57:03 dist-test-slave-x1lh krb5kdc[1796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377023, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.252@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
Sep 25 22:57:03 dist-test-slave-x1lh krb5kdc[1796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.252@KRBTEST.COM
I20260925 22:57:03.440891 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.425102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54574 (local address 127.20.114.254:36469)
0925 22:57:03.425249 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:03.425255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:03.426656 (+ 1401us) server_negotiation.cc:407] Connection header received
0925 22:57:03.428136 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:03.428140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:03.428208 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:03.428321 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:03.429750 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.430566 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.432207 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.432575 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.432698 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:03.436301 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:03.436305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:03.436307 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:03.436339 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:03.438081 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.438709 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.438718 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.438719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.438802 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.439197 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.439200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.439202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.439421 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:03.439726 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:03.440445 (+ 719us) server_negotiation.cc:299] Negotiation successful
0925 22:57:03.440603 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:57:03.442209 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.425665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:36469 (local address 127.0.0.1:54574)
0925 22:57:03.426456 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.426535 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:03.427837 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:03.428442 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:03.428451 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:03.429154 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:03.429548 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:03.429590 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.430711 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.430714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:03.432017 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:03.432028 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.432770 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.432773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:03.433100 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:03.433896 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:03.433920 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:03.435958 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:03.438216 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:03.438227 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:03.438239 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:03.438583 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:03.438904 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:03.438907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:03.438909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:03.439085 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:03.439719 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:03.439727 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:03.440083 (+ 356us) client_negotiation.cc:769] Sending connection context
0925 22:57:03.440369 (+ 286us) client_negotiation.cc:240] Negotiation successful
0925 22:57:03.440933 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":184,"threads_started":1}
I20260925 22:57:03.442209 2031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.426500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40638 (local address 127.20.114.252:38453)
0925 22:57:03.426917 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:03.426930 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:03.427040 (+ 110us) server_negotiation.cc:407] Connection header received
0925 22:57:03.427859 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:03.427889 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:03.428299 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:03.428531 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:03.429541 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.430896 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.432201 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.432908 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.433039 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:03.436634 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:03.436642 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:03.436655 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:03.436680 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:03.438625 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.439306 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.439312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.439314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.439398 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.440254 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.440258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.440260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.440551 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:03.440787 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:03.441305 (+ 518us) server_negotiation.cc:299] Negotiation successful
0925 22:57:03.441459 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":119,"threads_started":1}
I20260925 22:57:03.444078 2027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:57:03 dist-test-slave-x1lh krb5kdc[1796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377023, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.252@KRBTEST.COM for kudu/127.20.114.253@KRBTEST.COM
Sep 25 22:57:03 dist-test-slave-x1lh krb5kdc[1796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:57:03.457695 1952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.445249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38924 (local address 127.20.114.253:33915)
0925 22:57:03.445531 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:03.445535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:03.445548 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:57:03.445593 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:03.445595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:03.445731 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:03.445825 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:03.446575 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.447189 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.449239 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.449626 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.449770 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:03.453306 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:03.453312 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:03.453314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:03.453354 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:03.455239 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.455865 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.455869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.455871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.455976 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.456447 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.456451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.456453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.456757 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:03.456844 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:03.457332 (+ 488us) server_negotiation.cc:299] Negotiation successful
0925 22:57:03.457503 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260925 22:57:03.457744 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.445089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:33915 (local address 127.0.0.1:38924)
0925 22:57:03.445263 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.445280 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:03.445386 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:03.445883 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:03.445887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:03.446240 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:03.446406 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:03.446432 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.447457 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.447460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:03.449012 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:03.449026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.449789 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.449792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:03.450026 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:03.450685 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:03.450704 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:03.453060 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:03.455407 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:03.455411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:03.455413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:03.455729 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:03.456161 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:03.456174 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:03.456177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:03.456307 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:03.456909 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:03.456913 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:03.457096 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 22:57:03.457221 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 22:57:03.457575 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 22:57:03.457948 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.446011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54590 (local address 127.20.114.254:36469)
0925 22:57:03.446125 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:03.446130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:03.446205 (+ 75us) server_negotiation.cc:407] Connection header received
0925 22:57:03.446330 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:03.446332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:03.446373 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:03.446439 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:03.447117 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.447677 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.449435 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.449815 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.449945 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:03.453549 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:03.453553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:03.453555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:03.453590 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:03.455460 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.456162 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.456167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.456169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.456300 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.456733 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.456736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.456737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.457046 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:03.457297 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:03.457615 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 22:57:03.457751 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 22:57:03.459265 2027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:57:03 dist-test-slave-x1lh krb5kdc[1796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377023, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.252@KRBTEST.COM for kudu/127.20.114.252@KRBTEST.COM
Sep 25 22:57:03 dist-test-slave-x1lh krb5kdc[1796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.253@KRBTEST.COM
I20260925 22:57:03.474366 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.459838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:38453 (local address 127.0.0.1:40644)
0925 22:57:03.460063 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.460077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:03.460144 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:03.460459 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:03.460463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:03.460750 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:03.460929 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:03.460939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.462402 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.462405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:03.463326 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:03.463339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.464476 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.464480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:03.464680 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:03.466386 (+ 1706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:03.466406 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:03.469234 (+ 2828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:03.472240 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:03.472245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:03.472248 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:03.472599 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:03.473004 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:03.473008 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:03.473010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:03.473142 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:03.473697 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:03.473700 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:03.473813 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 22:57:03.473941 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 22:57:03.474125 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260925 22:57:03.474414 2031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.459897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40644 (local address 127.20.114.252:38453)
0925 22:57:03.460164 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:03.460169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:03.460183 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:57:03.460227 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:03.460230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:03.460275 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:03.460363 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:03.461148 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.462137 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.463683 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.464079 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.464322 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:03.469488 (+ 5166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:03.469492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:03.469494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:03.469530 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:03.472054 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.472720 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.472724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.472728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.472817 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.473239 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.473242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.473244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.473580 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:03.473694 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:03.473999 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 22:57:03.474224 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":196}
I20260925 22:57:03.475505 1952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.460039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38928 (local address 127.20.114.253:33915)
0925 22:57:03.460150 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:03.460155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:03.460171 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:57:03.460232 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:03.460235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:03.460275 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:03.460363 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:03.461296 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.462148 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.463679 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.464067 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.464339 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:03.470069 (+ 5730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:03.470073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:03.470075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:03.470109 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:03.472442 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.473192 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.473199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.473204 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.473339 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.473702 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.473705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.473707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.473982 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:03.474287 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:03.475257 (+ 970us) server_negotiation.cc:299] Negotiation successful
0925 22:57:03.475388 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 22:57:03.479799 2027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9: Bootstrap starting.
I20260925 22:57:03.480260 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:57:03.484294 2027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9: Neither blocks nor log segments found. Creating new log.
I20260925 22:57:03.485817 2027 log.cc:824] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:57:03.488390 2027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9: No bootstrap required, opened a new log
I20260925 22:57:03.493613 2027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a64b4a9f5f444f7985ff1943f13a52b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36469 } } peers { permanent_uuid: "7fd052b7bbad4cad9610a68b0cd19e13" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } } peers { permanent_uuid: "234806cba37e458e89748000421701f9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } }
I20260925 22:57:03.493968 2027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:57:03.494033 2027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 234806cba37e458e89748000421701f9, State: Initialized, Role: FOLLOWER
I20260925 22:57:03.494658 2027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a64b4a9f5f444f7985ff1943f13a52b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36469 } } peers { permanent_uuid: "7fd052b7bbad4cad9610a68b0cd19e13" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } } peers { permanent_uuid: "234806cba37e458e89748000421701f9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } }
I20260925 22:57:03.496743 1952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.483985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38932 (local address 127.20.114.253:33915)
0925 22:57:03.484508 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:03.484514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:03.485107 (+ 593us) server_negotiation.cc:407] Connection header received
0925 22:57:03.485423 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:03.485427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:03.485487 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:03.485533 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:03.486663 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.487466 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.489196 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.489568 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.489654 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:03.492252 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:03.492256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:03.492259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:03.492291 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:03.494502 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.495202 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.495206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.495208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.495320 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.495780 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.495783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.495785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.496028 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:03.496121 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:03.496440 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 22:57:03.496563 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324}
I20260925 22:57:03.496776 2027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:57:03.497128 2031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.484336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40658 (local address 127.20.114.252:38453)
0925 22:57:03.484495 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:03.484500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:03.485119 (+ 619us) server_negotiation.cc:407] Connection header received
0925 22:57:03.485638 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:03.485643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:03.485706 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:03.485799 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:03.486663 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.487466 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.489196 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.489574 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.489654 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:03.492199 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:03.492205 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:03.492208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:03.492241 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:03.494509 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.495202 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.495206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.495208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.495320 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.495640 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.495644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.495646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.495928 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:03.496059 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:03.496818 (+ 759us) server_negotiation.cc:299] Negotiation successful
0925 22:57:03.496981 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 22:57:03.498906 2040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a64b4a9f5f444f7985ff1943f13a52b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36469 } } peers { permanent_uuid: "7fd052b7bbad4cad9610a68b0cd19e13" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } } peers { permanent_uuid: "234806cba37e458e89748000421701f9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } } }
I20260925 22:57:03.499079 2040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:57:03.501062 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.484204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54592 (local address 127.20.114.254:36469)
0925 22:57:03.484377 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:03.484383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:03.486690 (+ 2307us) server_negotiation.cc:407] Connection header received
0925 22:57:03.489048 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:03.489053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:03.489201 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:03.489267 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:03.490943 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.491729 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.493849 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.494204 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.494295 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:03.496461 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:03.496465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:03.496467 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:03.496502 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:03.498935 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.499546 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.499550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.499552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.499660 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.500027 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.500032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.500034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.500301 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:03.500456 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:03.500729 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 22:57:03.500903 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260925 22:57:03.503057 2051 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:57:03.503154 2051 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:57:03.501078 1959 master_runner.cc:186] Error getting master registration for 127.20.114.254:36469: OK, instance_id { permanent_uuid: "0a64b4a9f5f444f7985ff1943f13a52b" instance_seqno: 1790377022903300 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:57:03.505118 1959 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:57:03.553186 1877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } attempt: 4
I20260925 22:57:03.554031 1953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } attempt: 4
Sep 25 22:57:03 dist-test-slave-x1lh krb5kdc[1796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377023, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.253@KRBTEST.COM for kudu/127.20.114.252@KRBTEST.COM
Sep 25 22:57:03 dist-test-slave-x1lh krb5kdc[1796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377022, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for kudu/127.20.114.252@KRBTEST.COM
I20260925 22:57:03.564618 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.554456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:38453 (local address 127.0.0.1:40684)
0925 22:57:03.554649 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.554663 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:03.554747 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:03.555288 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:03.555291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:03.555483 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:03.555596 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:03.555603 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.556456 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.556460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:03.557134 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:03.557145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.557799 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.557805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:03.557922 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:03.558570 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:03.558586 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:03.560284 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:03.562823 (+ 2539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:03.562827 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:03.562830 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:03.563121 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:03.563434 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:03.563438 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:03.563440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:03.563543 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:03.564033 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:03.564036 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:03.564162 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 22:57:03.564271 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 22:57:03.564435 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260925 22:57:03.564685 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.554585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40684 (local address 127.20.114.252:38453)
0925 22:57:03.555006 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:03.555012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:03.555023 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:57:03.555108 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:03.555110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:03.555145 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:03.555258 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:03.555732 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.556325 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.557276 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.557665 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.557775 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:03.560435 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:03.560439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:03.560442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:03.560476 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:03.562670 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.563236 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.563239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.563242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.563332 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.563650 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.563656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.563661 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.563923 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:03.564030 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:03.564342 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 22:57:03.564473 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":157,"threads_started":1}
I20260925 22:57:03.565451 2031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.553971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40668 (local address 127.20.114.252:38453)
0925 22:57:03.554162 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:03.554167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:03.554314 (+ 147us) server_negotiation.cc:407] Connection header received
0925 22:57:03.554428 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:03.554432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:03.554485 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:03.554618 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:03.555249 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.555880 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.557083 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.557388 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.557510 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:03.561524 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:03.561528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:03.561534 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:03.561564 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:03.563149 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.563706 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.563710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.563712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.563796 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:03.564103 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:03.564106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:03.564108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:03.564294 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:03.564381 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:03.565201 (+ 820us) server_negotiation.cc:299] Negotiation successful
0925 22:57:03.565324 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 22:57:03.565914 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:03.553849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:38453 (local address 127.0.0.1:40668)
0925 22:57:03.554172 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:03.554190 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:03.554311 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:03.554749 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:03.554752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:03.554976 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:03.555108 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:03.555115 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.556012 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.556016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:03.556944 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:03.556958 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:03.557522 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:03.557533 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:03.557632 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:03.559558 (+ 1926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:03.559573 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:03.561346 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:03.563291 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:03.563297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:03.563300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:03.563590 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:03.563892 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:03.563895 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:03.563897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:03.564000 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:03.564428 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:03.564434 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:03.564538 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 22:57:03.564649 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 22:57:03.565725 (+ 1076us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223}
I20260925 22:57:03.567673 1953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fd052b7bbad4cad9610a68b0cd19e13: Bootstrap starting.
I20260925 22:57:03.571782 1953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fd052b7bbad4cad9610a68b0cd19e13: Neither blocks nor log segments found. Creating new log.
I20260925 22:57:03.572374 1877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a64b4a9f5f444f7985ff1943f13a52b: Bootstrap starting.
I20260925 22:57:03.572520 1953 log.cc:824] T 00000000000000000000000000000000 P 7fd052b7bbad4cad9610a68b0cd19e13: Log is configured to *not* fsync() on all Append() calls
I20260925 22:57:03.574532 1953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fd052b7bbad4cad9610a68b0cd19e13: No bootstrap required, opened a new log
I20260925 22:57:03.575722 1877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a64b4a9f5f444f7985ff1943f13a52b: Neither blocks nor log segments found. Creating new log.
I20260925 22:57:03.576709 1877 log.cc:824] T 00000000000000000000000000000000 P 0a64b4a9f5f444f7985ff1943f13a52b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:57:03.578038 1953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fd052b7bbad4cad9610a68b0cd19e13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a64b4a9f5f444f7985ff1943f13a52b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36469 } } peers { permanent_uuid: "7fd052b7bbad4cad9610a68b0cd19e13" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } } peers { permanent_uuid: "234806cba37e458e89748000421701f9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } }
I20260925 22:57:03.578326 1953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fd052b7bbad4cad9610a68b0cd19e13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:57:03.578392 1953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fd052b7bbad4cad9610a68b0cd19e13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd052b7bbad4cad9610a68b0cd19e13, State: Initialized, Role: FOLLOWER
I20260925 22:57:03.578903 1953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fd052b7bbad4cad9610a68b0cd19e13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a64b4a9f5f444f7985ff1943f13a52b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36469 } } peers { permanent_uuid: "7fd052b7bbad4cad9610a68b0cd19e13" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } } peers { permanent_uuid: "234806cba37e458e89748000421701f9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } }
I20260925 22:57:03.579401 1877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a64b4a9f5f444f7985ff1943f13a52b: No bootstrap required, opened a new log
I20260925 22:57:03.579754 2040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:57:03.579913 2040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a64b4a9f5f444f7985ff1943f13a52b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36469 } } peers { permanent_uuid: "7fd052b7bbad4cad9610a68b0cd19e13" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } } peers { permanent_uuid: "234806cba37e458e89748000421701f9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } }
I20260925 22:57:03.581043 2056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fd052b7bbad4cad9610a68b0cd19e13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a64b4a9f5f444f7985ff1943f13a52b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36469 } } peers { permanent_uuid: "7fd052b7bbad4cad9610a68b0cd19e13" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } } peers { permanent_uuid: "234806cba37e458e89748000421701f9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } } }
I20260925 22:57:03.581259 2056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fd052b7bbad4cad9610a68b0cd19e13 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:57:03.581328 1953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fd052b7bbad4cad9610a68b0cd19e13 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:57:03.581749 2040 leader_election.cc:288] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a64b4a9f5f444f7985ff1943f13a52b (127.20.114.254:36469), 7fd052b7bbad4cad9610a68b0cd19e13 (127.20.114.253:33915)
I20260925 22:57:03.582526 1926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "234806cba37e458e89748000421701f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd052b7bbad4cad9610a68b0cd19e13" is_pre_election: true
I20260925 22:57:03.582856 1926 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7fd052b7bbad4cad9610a68b0cd19e13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 234806cba37e458e89748000421701f9 in term 0.
I20260925 22:57:03.583343 1978 leader_election.cc:302] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 234806cba37e458e89748000421701f9, 7fd052b7bbad4cad9610a68b0cd19e13; no voters:
I20260925 22:57:03.583679 2040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:57:03.583765 2040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:57:03.583817 2040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:57:03.585361 2040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a64b4a9f5f444f7985ff1943f13a52b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36469 } } peers { permanent_uuid: "7fd052b7bbad4cad9610a68b0cd19e13" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } } peers { permanent_uuid: "234806cba37e458e89748000421701f9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } }
I20260925 22:57:03.586736 1926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "234806cba37e458e89748000421701f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd052b7bbad4cad9610a68b0cd19e13"
I20260925 22:57:03.586930 1926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fd052b7bbad4cad9610a68b0cd19e13 [term 0 FOLLOWER]: Advancing to term 1
W20260925 22:57:03.587651 2067 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7fd052b7bbad4cad9610a68b0cd19e13: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:57:03.587736 2067 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:57:03.588392 1926 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7fd052b7bbad4cad9610a68b0cd19e13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 234806cba37e458e89748000421701f9 in term 1.
I20260925 22:57:03.588800 1978 leader_election.cc:302] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 234806cba37e458e89748000421701f9, 7fd052b7bbad4cad9610a68b0cd19e13; no voters:
I20260925 22:57:03.586270 2040 leader_election.cc:288] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [CANDIDATE]: Term 1 election: Requested vote from peers 0a64b4a9f5f444f7985ff1943f13a52b (127.20.114.254:36469), 7fd052b7bbad4cad9610a68b0cd19e13 (127.20.114.253:33915)
I20260925 22:57:03.589393 2069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:57:03.585256 1884 master_runner.cc:186] Error getting master registration for 127.20.114.254:36469: OK, instance_id { permanent_uuid: "0a64b4a9f5f444f7985ff1943f13a52b" instance_seqno: 1790377022903300 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:57:03.589687 1884 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:57:03.589771 2069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [term 1 LEADER]: Becoming Leader. State: Replica: 234806cba37e458e89748000421701f9, State: Running, Role: LEADER
I20260925 22:57:03.590513 2069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a64b4a9f5f444f7985ff1943f13a52b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36469 } } peers { permanent_uuid: "7fd052b7bbad4cad9610a68b0cd19e13" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } } peers { permanent_uuid: "234806cba37e458e89748000421701f9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } }
I20260925 22:57:03.596436 2040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 234806cba37e458e89748000421701f9. Latest consensus state: current_term: 1 leader_uuid: "234806cba37e458e89748000421701f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a64b4a9f5f444f7985ff1943f13a52b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36469 } } peers { permanent_uuid: "7fd052b7bbad4cad9610a68b0cd19e13" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } } peers { permanent_uuid: "234806cba37e458e89748000421701f9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } } }
I20260925 22:57:03.596613 2040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:57:03.598312 1877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a64b4a9f5f444f7985ff1943f13a52b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a64b4a9f5f444f7985ff1943f13a52b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36469 } } peers { permanent_uuid: "7fd052b7bbad4cad9610a68b0cd19e13" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } } peers { permanent_uuid: "234806cba37e458e89748000421701f9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } }
I20260925 22:57:03.598642 1877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a64b4a9f5f444f7985ff1943f13a52b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:57:03.598714 1877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a64b4a9f5f444f7985ff1943f13a52b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a64b4a9f5f444f7985ff1943f13a52b, State: Initialized, Role: FOLLOWER
I20260925 22:57:03.599484 1877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a64b4a9f5f444f7985ff1943f13a52b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a64b4a9f5f444f7985ff1943f13a52b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36469 } } peers { permanent_uuid: "7fd052b7bbad4cad9610a68b0cd19e13" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } } peers { permanent_uuid: "234806cba37e458e89748000421701f9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } }
I20260925 22:57:03.600238 2072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a64b4a9f5f444f7985ff1943f13a52b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a64b4a9f5f444f7985ff1943f13a52b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36469 } } peers { permanent_uuid: "7fd052b7bbad4cad9610a68b0cd19e13" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } } peers { permanent_uuid: "234806cba37e458e89748000421701f9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } } }
I20260925 22:57:03.600597 2072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a64b4a9f5f444f7985ff1943f13a52b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:57:03.601169 1877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a64b4a9f5f444f7985ff1943f13a52b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:57:03.601223 1850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "234806cba37e458e89748000421701f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a64b4a9f5f444f7985ff1943f13a52b" is_pre_election: true
I20260925 22:57:03.601550 1850 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0a64b4a9f5f444f7985ff1943f13a52b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 234806cba37e458e89748000421701f9 in term 0.
I20260925 22:57:03.601951 1851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "234806cba37e458e89748000421701f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a64b4a9f5f444f7985ff1943f13a52b"
I20260925 22:57:03.602070 1851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a64b4a9f5f444f7985ff1943f13a52b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:57:03.604082 1851 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0a64b4a9f5f444f7985ff1943f13a52b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 234806cba37e458e89748000421701f9 in term 1.
W20260925 22:57:03.608903 2083 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a64b4a9f5f444f7985ff1943f13a52b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:57:03.609094 2083 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:57:03.609193 2071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:57:03.610205 2071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:57:03.619640 1926 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7fd052b7bbad4cad9610a68b0cd19e13 [term 1 FOLLOWER]: Refusing update from remote peer 234806cba37e458e89748000421701f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:57:03.620636 2069 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd052b7bbad4cad9610a68b0cd19e13" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:57:03.625300 2056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fd052b7bbad4cad9610a68b0cd19e13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 234806cba37e458e89748000421701f9. Latest consensus state: current_term: 1 leader_uuid: "234806cba37e458e89748000421701f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a64b4a9f5f444f7985ff1943f13a52b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36469 } } peers { permanent_uuid: "7fd052b7bbad4cad9610a68b0cd19e13" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } } peers { permanent_uuid: "234806cba37e458e89748000421701f9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } } }
I20260925 22:57:03.625484 2056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fd052b7bbad4cad9610a68b0cd19e13 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:57:03.626842 1851 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0a64b4a9f5f444f7985ff1943f13a52b [term 1 FOLLOWER]: Refusing update from remote peer 234806cba37e458e89748000421701f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:57:03.627805 2069 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a64b4a9f5f444f7985ff1943f13a52b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:57:03.636516 2072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a64b4a9f5f444f7985ff1943f13a52b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 234806cba37e458e89748000421701f9. Latest consensus state: current_term: 1 leader_uuid: "234806cba37e458e89748000421701f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a64b4a9f5f444f7985ff1943f13a52b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36469 } } peers { permanent_uuid: "7fd052b7bbad4cad9610a68b0cd19e13" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } } peers { permanent_uuid: "234806cba37e458e89748000421701f9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } } }
I20260925 22:57:03.636672 2072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a64b4a9f5f444f7985ff1943f13a52b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:57:03.638568 2069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "234806cba37e458e89748000421701f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a64b4a9f5f444f7985ff1943f13a52b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36469 } } peers { permanent_uuid: "7fd052b7bbad4cad9610a68b0cd19e13" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } } peers { permanent_uuid: "234806cba37e458e89748000421701f9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } } }
I20260925 22:57:03.638728 2069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:57:03.640209 2056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fd052b7bbad4cad9610a68b0cd19e13 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "234806cba37e458e89748000421701f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a64b4a9f5f444f7985ff1943f13a52b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36469 } } peers { permanent_uuid: "7fd052b7bbad4cad9610a68b0cd19e13" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } } peers { permanent_uuid: "234806cba37e458e89748000421701f9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } } }
I20260925 22:57:03.640367 2056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fd052b7bbad4cad9610a68b0cd19e13 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:57:03.641012 2085 mvcc.cc:205] Tried to move back new op lower bound from 7333384288737972224 to 7333384288633524224. Current Snapshot: MvccSnapshot[applied={T|T < 7333384288737972224}]
I20260925 22:57:03.642709 2086 mvcc.cc:205] Tried to move back new op lower bound from 7333384288737972224 to 7333384288633524224. Current Snapshot: MvccSnapshot[applied={T|T < 7333384288737972224}]
I20260925 22:57:03.642899 2071 catalog_manager.cc:1347] Generated new cluster ID: 44b4b63beb6d4a7885d23573f11b7278
I20260925 22:57:03.642982 2071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:57:03.645439 2040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "234806cba37e458e89748000421701f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a64b4a9f5f444f7985ff1943f13a52b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36469 } } peers { permanent_uuid: "7fd052b7bbad4cad9610a68b0cd19e13" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } } peers { permanent_uuid: "234806cba37e458e89748000421701f9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } } }
I20260925 22:57:03.645605 2040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:57:03.646956 2072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a64b4a9f5f444f7985ff1943f13a52b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "234806cba37e458e89748000421701f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a64b4a9f5f444f7985ff1943f13a52b" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 36469 } } peers { permanent_uuid: "7fd052b7bbad4cad9610a68b0cd19e13" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 33915 } } peers { permanent_uuid: "234806cba37e458e89748000421701f9" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 38453 } } }
I20260925 22:57:03.647109 2072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a64b4a9f5f444f7985ff1943f13a52b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:57:03.661923 2071 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:57:03.663551 2071 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:57:03.677096 2071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 234806cba37e458e89748000421701f9: Generated new TSK 0
I20260925 22:57:03.678208 2071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:57:04.589452 2067 catalog_manager.cc:1259] Loaded cluster ID: 44b4b63beb6d4a7885d23573f11b7278
I20260925 22:57:04.589553 2067 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7fd052b7bbad4cad9610a68b0cd19e13: loading cluster ID for follower catalog manager: success
I20260925 22:57:04.591425 2067 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7fd052b7bbad4cad9610a68b0cd19e13: acquiring CA information for follower catalog manager: success
I20260925 22:57:04.592891 2067 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7fd052b7bbad4cad9610a68b0cd19e13: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:57:04.610816 2083 catalog_manager.cc:1259] Loaded cluster ID: 44b4b63beb6d4a7885d23573f11b7278
I20260925 22:57:04.610920 2083 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0a64b4a9f5f444f7985ff1943f13a52b: loading cluster ID for follower catalog manager: success
I20260925 22:57:04.612772 2083 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0a64b4a9f5f444f7985ff1943f13a52b: acquiring CA information for follower catalog manager: success
I20260925 22:57:04.614221 2083 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0a64b4a9f5f444f7985ff1943f13a52b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:57:04.759212 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 1809
I20260925 22:57:04.765941 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 1884
I20260925 22:57:04.772197 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 1959
2026-09-25T22:57:04Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3306 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:57:04 dist-test-slave-x1lh krb5kdc[2100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:57:04 dist-test-slave-x1lh krb5kdc[2100](info): set up 2 sockets
Sep 25 22:57:04 dist-test-slave-x1lh krb5kdc[2100](info): commencing operation
krb5kdc: starting...
W20260925 22:57:05.889879 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:57:05 dist-test-slave-x1lh krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:57:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:57:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:57:06.057312 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:34855
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:43971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:34855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:57:06.185698 2113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:06.186076 2113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:06.186156 2113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:06.190527 2113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:57:06.190639 2113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:06.190678 2113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:06.190740 2113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:57:06.190798 2113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:57:06.197045 2113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:43971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:34855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:34855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:06.198258 2113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:06.199378 2113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:57:06.209723 2121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:06.209818 2118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:06.209946 2113 server_base.cc:1034] running on GCE node
W20260925 22:57:06.209805 2119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:06.210602 2113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:06.212137 2113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:06.213358 2113 hybrid_clock.cc:648] HybridClock initialized: now 1790377026213338 us; error 39 us; skew 500 ppm
Sep 25 22:57:06 dist-test-slave-x1lh krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377026, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:06.216696 2113 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:57:06.217829 2113 webserver.cc:460] Webserver started at http://127.20.114.254:44063/ using document root <none> and password file <none>
I20260925 22:57:06.218475 2113 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:06.218586 2113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:06.218837 2113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:57:06.221647 2113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "06d061ff5b754c7d8f8d194c7518045e"
format_stamp: "Formatted at 2026-09-25 22:57:06 on dist-test-slave-x1lh"
server_key: "c99c2002a9c42d126e2fbf898500029e"
server_key_iv: "f524a2a79513ad2d2c81fbf8079324dc"
server_key_version: "encryptionkey@0"
I20260925 22:57:06.222365 2113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "06d061ff5b754c7d8f8d194c7518045e"
format_stamp: "Formatted at 2026-09-25 22:57:06 on dist-test-slave-x1lh"
server_key: "c99c2002a9c42d126e2fbf898500029e"
server_key_iv: "f524a2a79513ad2d2c81fbf8079324dc"
server_key_version: "encryptionkey@0"
I20260925 22:57:06.227522 2113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 22:57:06.230713 2128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:06.232424 2113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:57:06.232591 2113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "06d061ff5b754c7d8f8d194c7518045e"
format_stamp: "Formatted at 2026-09-25 22:57:06 on dist-test-slave-x1lh"
server_key: "c99c2002a9c42d126e2fbf898500029e"
server_key_iv: "f524a2a79513ad2d2c81fbf8079324dc"
server_key_version: "encryptionkey@0"
I20260925 22:57:06.232722 2113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:06.247855 2113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:06.252236 2113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:06.252426 2113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:06.261250 2113 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:34855
I20260925 22:57:06.261307 2180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:34855 every 8 connection(s)
I20260925 22:57:06.262492 2113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:57:06.265736 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 2113
I20260925 22:57:06.266135 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:57:06.266402 2181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:57:06.266691 20939 external_mini_cluster.cc:1428] Setting key e3b60a2883ee0738440595a3af2a28b4
I20260925 22:57:06.275679 2181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d061ff5b754c7d8f8d194c7518045e: Bootstrap starting.
Sep 25 22:57:06 dist-test-slave-x1lh krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:57:06.279027 2181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06d061ff5b754c7d8f8d194c7518045e: Neither blocks nor log segments found. Creating new log.
I20260925 22:57:06.280481 2181 log.cc:824] T 00000000000000000000000000000000 P 06d061ff5b754c7d8f8d194c7518045e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:57:06.283491 2181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d061ff5b754c7d8f8d194c7518045e: No bootstrap required, opened a new log
I20260925 22:57:06.286958 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:06.268456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54170 (local address 127.20.114.254:34855)
0925 22:57:06.269164 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:06.269185 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:06.269235 (+ 50us) server_negotiation.cc:407] Connection header received
0925 22:57:06.270177 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:06.270204 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:06.270708 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:06.271199 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:06.272238 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:06.273826 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:06.274965 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:06.275720 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:06.275900 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:06.280016 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:06.280024 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:06.280046 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:06.280102 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:06.282881 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:06.283716 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:06.283730 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:06.283736 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:06.283874 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:06.284324 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:06.284327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:06.284328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:06.284957 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:06.285119 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:06.285536 (+ 417us) server_negotiation.cc:299] Negotiation successful
0925 22:57:06.286057 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":146,"threads_started":1}
I20260925 22:57:06.288054 2181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06d061ff5b754c7d8f8d194c7518045e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d061ff5b754c7d8f8d194c7518045e" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34855 } }
I20260925 22:57:06.288313 2181 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06d061ff5b754c7d8f8d194c7518045e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:57:06.288391 2181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06d061ff5b754c7d8f8d194c7518045e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d061ff5b754c7d8f8d194c7518045e, State: Initialized, Role: FOLLOWER
I20260925 22:57:06.288968 2181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06d061ff5b754c7d8f8d194c7518045e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d061ff5b754c7d8f8d194c7518045e" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34855 } }
I20260925 22:57:06.289158 2181 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06d061ff5b754c7d8f8d194c7518045e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:57:06.289233 2181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06d061ff5b754c7d8f8d194c7518045e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:57:06.289337 2181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06d061ff5b754c7d8f8d194c7518045e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:57:06.290753 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:57:06.290925 2181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06d061ff5b754c7d8f8d194c7518045e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d061ff5b754c7d8f8d194c7518045e" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34855 } }
I20260925 22:57:06.291519 2181 leader_election.cc:302] T 00000000000000000000000000000000 P 06d061ff5b754c7d8f8d194c7518045e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06d061ff5b754c7d8f8d194c7518045e; no voters:
I20260925 22:57:06.292582 2181 leader_election.cc:288] T 00000000000000000000000000000000 P 06d061ff5b754c7d8f8d194c7518045e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:57:06.292591 2186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06d061ff5b754c7d8f8d194c7518045e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:57:06.292939 2186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06d061ff5b754c7d8f8d194c7518045e [term 1 LEADER]: Becoming Leader. State: Replica: 06d061ff5b754c7d8f8d194c7518045e, State: Running, Role: LEADER
I20260925 22:57:06.293332 2186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06d061ff5b754c7d8f8d194c7518045e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d061ff5b754c7d8f8d194c7518045e" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34855 } }
I20260925 22:57:06.295280 2181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06d061ff5b754c7d8f8d194c7518045e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:57:06.295264 2188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d061ff5b754c7d8f8d194c7518045e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06d061ff5b754c7d8f8d194c7518045e. Latest consensus state: current_term: 1 leader_uuid: "06d061ff5b754c7d8f8d194c7518045e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d061ff5b754c7d8f8d194c7518045e" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34855 } } }
I20260925 22:57:06.295579 2188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d061ff5b754c7d8f8d194c7518045e [sys.catalog]: This master's current role is: LEADER
I20260925 22:57:06.295416 2187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d061ff5b754c7d8f8d194c7518045e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06d061ff5b754c7d8f8d194c7518045e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d061ff5b754c7d8f8d194c7518045e" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34855 } } }
I20260925 22:57:06.295733 2187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d061ff5b754c7d8f8d194c7518045e [sys.catalog]: This master's current role is: LEADER
I20260925 22:57:06.296218 2190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:57:06.300091 2190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:57:06.307439 2190 catalog_manager.cc:1347] Generated new cluster ID: 709d9165d6d74bdeaecbb4d806298a21
I20260925 22:57:06.307562 2190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:57:06.333411 2190 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:57:06.334563 2190 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:57:06.344790 2190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06d061ff5b754c7d8f8d194c7518045e: Generated new TSK 0
I20260925 22:57:06.345870 2190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:57:06.399571 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 2113
2026-09-25T22:57:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1624 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:57:06 dist-test-slave-x1lh krb5kdc[2208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:57:06 dist-test-slave-x1lh krb5kdc[2208](info): set up 2 sockets
Sep 25 22:57:06 dist-test-slave-x1lh krb5kdc[2208](info): commencing operation
krb5kdc: starting...
W20260925 22:57:07.533780 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:57:07 dist-test-slave-x1lh krb5kdc[2208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:57:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:57:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:57:07.688112 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:43183
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:43183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:57:07.817528 2221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:07.817873 2221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:07.817981 2221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:07.822904 2221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:57:07.823006 2221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:07.823029 2221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:07.823048 2221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:57:07.823065 2221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:57:07.829767 2221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:43183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:43183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:07.831583 2221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:07.832914 2221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:57:07.843385 2227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:07.843385 2226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:07.843385 2229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:07.843688 2221 server_base.cc:1034] running on GCE node
I20260925 22:57:07.844743 2221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:07.846508 2221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:07.847741 2221 hybrid_clock.cc:648] HybridClock initialized: now 1790377027847656 us; error 107 us; skew 500 ppm
Sep 25 22:57:07 dist-test-slave-x1lh krb5kdc[2208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377027, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:07.851809 2221 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:57:07.853436 2221 webserver.cc:460] Webserver started at http://127.20.114.254:44623/ using document root <none> and password file <none>
I20260925 22:57:07.854115 2221 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:07.854226 2221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:07.854504 2221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:57:07.857301 2221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "d3c132d8718d473c94d4fe821ec671f2"
format_stamp: "Formatted at 2026-09-25 22:57:07 on dist-test-slave-x1lh"
server_key: "66b48594ac034fb968b6c47da26d1ed8"
server_key_iv: "1609fa952a5439cb8ceb8caf2f9a099e"
server_key_version: "encryptionkey@0"
I20260925 22:57:07.858127 2221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "d3c132d8718d473c94d4fe821ec671f2"
format_stamp: "Formatted at 2026-09-25 22:57:07 on dist-test-slave-x1lh"
server_key: "66b48594ac034fb968b6c47da26d1ed8"
server_key_iv: "1609fa952a5439cb8ceb8caf2f9a099e"
server_key_version: "encryptionkey@0"
I20260925 22:57:07.863116 2221 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 22:57:07.866878 2236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:07.868703 2221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:57:07.868943 2221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "d3c132d8718d473c94d4fe821ec671f2"
format_stamp: "Formatted at 2026-09-25 22:57:07 on dist-test-slave-x1lh"
server_key: "66b48594ac034fb968b6c47da26d1ed8"
server_key_iv: "1609fa952a5439cb8ceb8caf2f9a099e"
server_key_version: "encryptionkey@0"
I20260925 22:57:07.869112 2221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:07.894089 2221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:07.898895 2221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:07.899158 2221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:07.910061 2221 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:43183
I20260925 22:57:07.910601 2288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:43183 every 8 connection(s)
I20260925 22:57:07.911556 2221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:57:07.915031 2289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:57:07.917168 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 2221
I20260925 22:57:07.917302 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:57:07.917599 20939 external_mini_cluster.cc:1428] Setting key 4c9eafbe86296593429cee57884734f2
I20260925 22:57:07.922720 2289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2: Bootstrap starting.
I20260925 22:57:07.925711 2289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2: Neither blocks nor log segments found. Creating new log.
I20260925 22:57:07.926620 2289 log.cc:824] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:57:07 dist-test-slave-x1lh krb5kdc[2208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:57:07.929119 2289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2: No bootstrap required, opened a new log
I20260925 22:57:07.933776 2289 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c132d8718d473c94d4fe821ec671f2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43183 } }
I20260925 22:57:07.934100 2289 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:57:07.934194 2289 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3c132d8718d473c94d4fe821ec671f2, State: Initialized, Role: FOLLOWER
I20260925 22:57:07.934759 2289 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c132d8718d473c94d4fe821ec671f2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43183 } }
I20260925 22:57:07.934937 2289 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:57:07.935031 2289 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:57:07.935161 2289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:57:07.935493 2292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:07.919758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52910 (local address 127.20.114.254:43183)
0925 22:57:07.920565 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:07.920590 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:07.920633 (+ 43us) server_negotiation.cc:407] Connection header received
0925 22:57:07.921638 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:07.921660 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:07.922065 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:07.922477 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:07.923400 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:07.924452 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:07.925735 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:07.926349 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:07.926559 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:07.929404 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:07.929410 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:07.929426 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:07.929458 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:07.931761 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:07.932381 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:07.932387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:07.932393 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:07.932527 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:07.932858 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:07.932879 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:07.932881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:07.933401 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:07.933654 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:07.933955 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 22:57:07.934327 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":166,"threads_started":1}
I20260925 22:57:07.936401 2289 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c132d8718d473c94d4fe821ec671f2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43183 } }
I20260925 22:57:07.937000 2289 leader_election.cc:302] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3c132d8718d473c94d4fe821ec671f2; no voters:
I20260925 22:57:07.937404 2289 leader_election.cc:288] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:57:07.937505 2294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:57:07.937685 2294 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 1 LEADER]: Becoming Leader. State: Replica: d3c132d8718d473c94d4fe821ec671f2, State: Running, Role: LEADER
I20260925 22:57:07.938071 2294 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c132d8718d473c94d4fe821ec671f2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43183 } }
I20260925 22:57:07.938851 2289 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:57:07.940188 2296 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3c132d8718d473c94d4fe821ec671f2. Latest consensus state: current_term: 1 leader_uuid: "d3c132d8718d473c94d4fe821ec671f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c132d8718d473c94d4fe821ec671f2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43183 } } }
I20260925 22:57:07.940361 2296 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:57:07.940666 2295 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3c132d8718d473c94d4fe821ec671f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c132d8718d473c94d4fe821ec671f2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43183 } } }
I20260925 22:57:07.940783 2295 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:57:07.940960 2303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:57:07.944764 2303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:57:07.952039 2303 catalog_manager.cc:1347] Generated new cluster ID: 3512b03610ba46fab4de6dc9b45d6f14
I20260925 22:57:07.952142 2303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:57:07.984637 2303 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:57:07.986114 2303 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:57:07.992475 2303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2: Generated new TSK 0
I20260925 22:57:07.993579 2303 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.114.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
I20260925 22:57:08.049293 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:43183
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:57:08.190896 2315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:08.191263 2315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:08.191349 2315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:08.196271 2315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:08.196385 2315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:08.196488 2315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:57:08.202996 2315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:43183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:08.204299 2315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:08.205566 2315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:57:08.218582 2323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:08.218616 2321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:08.218765 2320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:08.219038 2315 server_base.cc:1034] running on GCE node
I20260925 22:57:08.219622 2315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:08.220418 2315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:08.221652 2315 hybrid_clock.cc:648] HybridClock initialized: now 1790377028221612 us; error 64 us; skew 500 ppm
Sep 25 22:57:08 dist-test-slave-x1lh krb5kdc[2208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377028, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:08.225204 2315 init.cc:377] Logged in from keytab as kudu/127.20.114.193@KRBTEST.COM (short username kudu)
I20260925 22:57:08.226296 2315 webserver.cc:460] Webserver started at http://127.20.114.193:41425/ using document root <none> and password file <none>
I20260925 22:57:08.226949 2315 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:08.227058 2315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:08.227305 2315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:57:08.229768 2315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "f07077db5dbd405d968bf7b0da05e2ff"
format_stamp: "Formatted at 2026-09-25 22:57:08 on dist-test-slave-x1lh"
server_key: "091208c7f52b0b69e7a6964c3e9cb66d"
server_key_iv: "3cf1ccd24414325ba917f4a7ea395ea1"
server_key_version: "encryptionkey@0"
I20260925 22:57:08.230454 2315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "f07077db5dbd405d968bf7b0da05e2ff"
format_stamp: "Formatted at 2026-09-25 22:57:08 on dist-test-slave-x1lh"
server_key: "091208c7f52b0b69e7a6964c3e9cb66d"
server_key_iv: "3cf1ccd24414325ba917f4a7ea395ea1"
server_key_version: "encryptionkey@0"
I20260925 22:57:08.235201 2315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:57:08.238628 2330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:08.241117 2315 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 22:57:08.241304 2315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "f07077db5dbd405d968bf7b0da05e2ff"
format_stamp: "Formatted at 2026-09-25 22:57:08 on dist-test-slave-x1lh"
server_key: "091208c7f52b0b69e7a6964c3e9cb66d"
server_key_iv: "3cf1ccd24414325ba917f4a7ea395ea1"
server_key_version: "encryptionkey@0"
I20260925 22:57:08.241458 2315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:08.268416 2315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:08.273032 2315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:08.273305 2315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:08.274065 2315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:57:08.275087 2315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:57:08.275148 2315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:08.275194 2315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:57:08.275210 2315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:08.286499 2315 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:39053
I20260925 22:57:08.286525 2443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:39053 every 8 connection(s)
I20260925 22:57:08.287629 2315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:57:08.290009 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 2315
I20260925 22:57:08.290205 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:57:08.290658 20939 external_mini_cluster.cc:1428] Setting key 233822eddf012143cd8cbc6614b69c47
Sep 25 22:57:08 dist-test-slave-x1lh krb5kdc[2208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377028, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.114.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.194@KRBTEST.COM" created.
I20260925 22:57:08.308250 2292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:08.290417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:39109 (local address 127.20.114.254:43183)
0925 22:57:08.290736 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:08.290743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:08.292257 (+ 1514us) server_negotiation.cc:407] Connection header received
0925 22:57:08.294688 (+ 2431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:08.294693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:08.294773 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:08.294894 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:08.296369 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.297013 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.298332 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.298585 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.298710 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:08.301902 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:08.301907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:08.301909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:08.301939 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:08.304237 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:08.305377 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:08.305386 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:08.305391 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:08.305527 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:08.306374 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:08.306378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:08.306379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:08.306617 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:08.306711 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:08.307737 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0925 22:57:08.307928 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260925 22:57:08.309226 2446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:08.291229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.193:39109)
0925 22:57:08.291879 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:08.292026 (+ 147us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:08.294283 (+ 2257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:08.295108 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:08.295117 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:08.295550 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:08.296102 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:08.296118 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.297176 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.297179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:08.298172 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:08.298185 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.298786 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.298792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:08.298988 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:08.299661 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:08.299680 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:08.301662 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:08.304411 (+ 2749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:08.304422 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:08.304447 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:08.305182 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:08.305688 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:08.305693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:08.305697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:08.306154 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:08.306752 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:08.306764 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:08.307275 (+ 511us) client_negotiation.cc:769] Sending connection context
0925 22:57:08.307558 (+ 283us) client_negotiation.cc:240] Negotiation successful
0925 22:57:08.307915 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":209,"threads_started":1}
I20260925 22:57:08.311208 2444 heartbeater.cc:347] Connected to a master server at 127.20.114.254:43183
I20260925 22:57:08.311560 2444 heartbeater.cc:464] Registering TS with master...
I20260925 22:57:08.312299 2444 heartbeater.cc:511] Master 127.20.114.254:43183 requested a full tablet report, sending...
I20260925 22:57:08.314659 2253 ts_manager.cc:194] Registered new tserver with Master: f07077db5dbd405d968bf7b0da05e2ff (127.20.114.193:39053)
I20260925 22:57:08.316107 2253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:39109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
I20260925 22:57:08.334184 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:43183
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:57:08.472468 2449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:08.472802 2449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:08.472851 2449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:08.477970 2449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:08.478070 2449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:08.478137 2449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:57:08.485719 2449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:43183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:08.486997 2449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:08.488184 2449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:57:08.503351 2449 server_base.cc:1034] running on GCE node
W20260925 22:57:08.503645 2457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:08.503567 2455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:08.503603 2454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:08.504256 2449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:08.505234 2449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:08.506500 2449 hybrid_clock.cc:648] HybridClock initialized: now 1790377028506445 us; error 64 us; skew 500 ppm
Sep 25 22:57:08 dist-test-slave-x1lh krb5kdc[2208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377028, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:08.511052 2449 init.cc:377] Logged in from keytab as kudu/127.20.114.194@KRBTEST.COM (short username kudu)
I20260925 22:57:08.512352 2449 webserver.cc:460] Webserver started at http://127.20.114.194:39915/ using document root <none> and password file <none>
I20260925 22:57:08.513273 2449 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:08.513367 2449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:08.513592 2449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:57:08.516348 2449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "956e81b9add74bad987c4aa1d19ab4e9"
format_stamp: "Formatted at 2026-09-25 22:57:08 on dist-test-slave-x1lh"
server_key: "0fe5227eaa09142190666b24a221c256"
server_key_iv: "16dfdc12d09536d911f45728e4bfd5f2"
server_key_version: "encryptionkey@0"
I20260925 22:57:08.517232 2449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "956e81b9add74bad987c4aa1d19ab4e9"
format_stamp: "Formatted at 2026-09-25 22:57:08 on dist-test-slave-x1lh"
server_key: "0fe5227eaa09142190666b24a221c256"
server_key_iv: "16dfdc12d09536d911f45728e4bfd5f2"
server_key_version: "encryptionkey@0"
I20260925 22:57:08.522166 2449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 22:57:08.525324 2464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:08.527216 2449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:57:08.527366 2449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "956e81b9add74bad987c4aa1d19ab4e9"
format_stamp: "Formatted at 2026-09-25 22:57:08 on dist-test-slave-x1lh"
server_key: "0fe5227eaa09142190666b24a221c256"
server_key_iv: "16dfdc12d09536d911f45728e4bfd5f2"
server_key_version: "encryptionkey@0"
I20260925 22:57:08.527479 2449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:08.575294 2449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:08.579216 2449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:08.579401 2449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:08.580081 2449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:57:08.581117 2449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:57:08.581176 2449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:08.581224 2449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:57:08.581239 2449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:08.594764 2449 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:45971
I20260925 22:57:08.594967 2577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:45971 every 8 connection(s)
I20260925 22:57:08.596288 2449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:57:08.604182 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 2449
I20260925 22:57:08.604321 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:57:08.604628 20939 external_mini_cluster.cc:1428] Setting key 25cf085480233e0bba4c410e880be87c
Sep 25 22:57:08 dist-test-slave-x1lh krb5kdc[2208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377028, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.114.195@KRBTEST.COM; defaulting to no policy
I20260925 22:57:08.616159 2292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:08.598971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:46873 (local address 127.20.114.254:43183)
0925 22:57:08.599134 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:08.599138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:08.600275 (+ 1137us) server_negotiation.cc:407] Connection header received
0925 22:57:08.601884 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:08.601887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:08.601945 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:08.602038 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:08.603623 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.604429 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.606355 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.606778 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.607025 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:08.610805 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:08.610809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:08.610811 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:08.610843 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:08.612823 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:08.613725 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:08.613729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:08.613731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:08.613836 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:08.614285 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:08.614289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:08.614291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:08.614817 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:08.615005 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:08.615817 (+ 812us) server_negotiation.cc:299] Negotiation successful
0925 22:57:08.615981 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:57:08.617226 2580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:08.599326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.194:46873)
0925 22:57:08.599955 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:08.600102 (+ 147us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:08.601528 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:08.602307 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:08.602316 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:08.602779 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:08.603363 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:08.603381 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.604660 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.604664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:08.606193 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:08.606209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.607086 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.607090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:08.607285 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:08.608182 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:08.608202 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:08.610538 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:08.613105 (+ 2567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:08.613113 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:08.613133 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:08.613600 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:08.613946 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:08.613949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:08.613951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:08.614157 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:08.614993 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:08.615001 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:08.615482 (+ 481us) client_negotiation.cc:769] Sending connection context
0925 22:57:08.615745 (+ 263us) client_negotiation.cc:240] Negotiation successful
0925 22:57:08.616173 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":159,"threads_started":1}
I20260925 22:57:08.619314 2578 heartbeater.cc:347] Connected to a master server at 127.20.114.254:43183
I20260925 22:57:08.620230 2578 heartbeater.cc:464] Registering TS with master...
I20260925 22:57:08.621292 2578 heartbeater.cc:511] Master 127.20.114.254:43183 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.195@KRBTEST.COM" created.
I20260925 22:57:08.623019 2253 ts_manager.cc:194] Registered new tserver with Master: 956e81b9add74bad987c4aa1d19ab4e9 (127.20.114.194:45971)
I20260925 22:57:08.623946 2253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:46873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
I20260925 22:57:08.657835 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:43183
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:57:08.803124 2583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:08.803550 2583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:08.803644 2583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:08.808369 2583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:08.808508 2583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:08.808625 2583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:57:08.814841 2583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:43183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:08.816284 2583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:08.817549 2583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:57:08.830741 2588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:08.830854 2591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:08.830741 2589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:08.830924 2583 server_base.cc:1034] running on GCE node
I20260925 22:57:08.832399 2583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:08.833336 2583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:08.836755 2583 hybrid_clock.cc:648] HybridClock initialized: now 1790377028836681 us; error 96 us; skew 500 ppm
Sep 25 22:57:08 dist-test-slave-x1lh krb5kdc[2208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377028, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:08.840685 2583 init.cc:377] Logged in from keytab as kudu/127.20.114.195@KRBTEST.COM (short username kudu)
I20260925 22:57:08.841895 2583 webserver.cc:460] Webserver started at http://127.20.114.195:34645/ using document root <none> and password file <none>
I20260925 22:57:08.842566 2583 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:08.842674 2583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:08.842931 2583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:57:08.846119 2583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "28ef18fda7e441508e538985b7fdc630"
format_stamp: "Formatted at 2026-09-25 22:57:08 on dist-test-slave-x1lh"
server_key: "2f59199e402207f35ca0849c8718ebdc"
server_key_iv: "9805d192c6518ec0a25a047058eb83b6"
server_key_version: "encryptionkey@0"
I20260925 22:57:08.846901 2583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "28ef18fda7e441508e538985b7fdc630"
format_stamp: "Formatted at 2026-09-25 22:57:08 on dist-test-slave-x1lh"
server_key: "2f59199e402207f35ca0849c8718ebdc"
server_key_iv: "9805d192c6518ec0a25a047058eb83b6"
server_key_version: "encryptionkey@0"
I20260925 22:57:08.852001 2583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 22:57:08.855396 2598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:08.857280 2583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:57:08.857467 2583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "28ef18fda7e441508e538985b7fdc630"
format_stamp: "Formatted at 2026-09-25 22:57:08 on dist-test-slave-x1lh"
server_key: "2f59199e402207f35ca0849c8718ebdc"
server_key_iv: "9805d192c6518ec0a25a047058eb83b6"
server_key_version: "encryptionkey@0"
I20260925 22:57:08.857620 2583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:08.890355 2583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:08.894991 2583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:08.895285 2583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:08.896050 2583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:57:08.897189 2583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:57:08.897277 2583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:08.897352 2583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:57:08.897401 2583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:08.910476 2583 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:43563
I20260925 22:57:08.910589 2711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:43563 every 8 connection(s)
I20260925 22:57:08.911857 2583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:57:08.918116 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 2583
I20260925 22:57:08.918262 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:57:08.918556 20939 external_mini_cluster.cc:1428] Setting key 057333b46a082dd9768aaeb6ad32c1f6
Sep 25 22:57:08 dist-test-slave-x1lh krb5kdc[2208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377028, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:57:08.929608 2292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:08.914325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:38093 (local address 127.20.114.254:43183)
0925 22:57:08.914474 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:08.914478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:08.915782 (+ 1304us) server_negotiation.cc:407] Connection header received
0925 22:57:08.917408 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:08.917411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:08.917479 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:08.917592 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:08.919388 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.920322 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.922138 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.922493 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.922604 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:08.925669 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:08.925673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:08.925676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:08.925715 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:08.927217 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:08.927833 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:08.927837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:08.927839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:08.927934 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:08.928303 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:08.928307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:08.928309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:08.928542 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:08.928659 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:08.929292 (+ 633us) server_negotiation.cc:299] Negotiation successful
0925 22:57:08.929438 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:57:08.930575 2714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:08.914872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.195:38093)
0925 22:57:08.915584 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:08.915649 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:08.917135 (+ 1486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:08.917819 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:08.917833 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:08.918425 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:08.919178 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:08.919200 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.920512 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.920515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:08.921996 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:08.922008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.922651 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.922655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:08.922851 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:08.923608 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:08.923625 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:08.925464 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:08.927373 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:08.927380 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:08.927399 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:08.927709 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:08.928033 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:08.928036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:08.928038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:08.928196 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:08.928658 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:08.928668 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:08.929017 (+ 349us) client_negotiation.cc:769] Sending connection context
0925 22:57:08.929228 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 22:57:08.929617 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":189,"threads_started":1}
I20260925 22:57:08.932184 2712 heartbeater.cc:347] Connected to a master server at 127.20.114.254:43183
I20260925 22:57:08.932508 2712 heartbeater.cc:464] Registering TS with master...
I20260925 22:57:08.933233 2712 heartbeater.cc:511] Master 127.20.114.254:43183 requested a full tablet report, sending...
I20260925 22:57:08.934597 2252 ts_manager.cc:194] Registered new tserver with Master: 28ef18fda7e441508e538985b7fdc630 (127.20.114.195:43563)
I20260925 22:57:08.935377 2252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:38093
I20260925 22:57:08.947794 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:57:08.961259 2292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:08.950884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52926 (local address 127.20.114.254:43183)
0925 22:57:08.951090 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:08.951100 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:08.951252 (+ 152us) server_negotiation.cc:407] Connection header received
0925 22:57:08.951419 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:08.951424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:08.951609 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:08.951720 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:08.952652 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.953315 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.954489 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.954749 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.954850 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:08.956826 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:08.956830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:08.956832 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:08.956863 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:08.958875 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:08.959464 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:08.959467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:08.959468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:08.959567 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:08.959975 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:08.959978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:08.959980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:08.960301 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:08.960439 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:08.960811 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 22:57:08.961083 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 22:57:08.966109 2252 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:57:08.970082 2252 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:57:08.989845 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:08.983704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39444 (local address 127.20.114.195:43563)
0925 22:57:08.984188 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:08.984201 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:08.984226 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:57:08.984339 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:08.984347 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:08.984584 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:08.984758 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:57:08.985326 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.986250 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.988186 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.988796 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.989202 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:08.989376 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:08.989546 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 22:57:08.989668 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":159,"threads_started":1}
I20260925 22:57:08.989840 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:08.982753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:43563 (local address 127.0.0.1:39444)
0925 22:57:08.983198 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:08.983244 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:08.983361 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:08.984822 (+ 1461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:08.984828 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:08.984855 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:57:08.985104 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:08.985114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.986398 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.986401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:08.988053 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:08.988063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.989016 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.989164 (+ 148us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:08.989295 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:08.989335 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 22:57:08.989488 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 22:57:08.989657 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":18,"thread_start_us":102,"threads_started":1}
I20260925 22:57:08.991333 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:08.983848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45448 (local address 127.20.114.193:39053)
0925 22:57:08.984667 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:08.984713 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:08.984744 (+ 31us) server_negotiation.cc:407] Connection header received
0925 22:57:08.984829 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:08.984834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:08.985108 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:08.985420 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:57:08.985758 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.986933 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.989203 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.990244 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.990357 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:08.990510 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:08.990867 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 22:57:08.990991 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":94,"threads_started":1}
I20260925 22:57:08.992071 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:08.982990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:39053 (local address 127.0.0.1:45448)
0925 22:57:08.984030 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:08.984047 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:08.984141 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:08.985465 (+ 1324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:08.985468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:08.985487 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:57:08.985577 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:08.985583 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.987323 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.987332 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:08.989040 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:08.989052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.990588 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.990595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:08.990702 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:08.990727 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:57:08.990812 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 22:57:08.990898 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":213,"thread_start_us":112,"threads_started":1}
I20260925 22:57:08.992344 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:08.982750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:45971 (local address 127.0.0.1:51368)
0925 22:57:08.984205 (+ 1455us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:08.984221 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:08.984323 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:08.984979 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:08.984981 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:08.984997 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:57:08.985305 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:08.985315 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.987334 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.987336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:08.989040 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:08.989052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.991813 (+ 2761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.991818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:08.991943 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:08.991970 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:57:08.992082 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 22:57:08.992180 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1382,"mutex_wait_us":1059,"thread_start_us":95,"threads_started":1}
I20260925 22:57:08.992969 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:08.982828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51368 (local address 127.20.114.194:45971)
0925 22:57:08.983885 (+ 1057us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:08.983898 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:08.984324 (+ 426us) server_negotiation.cc:407] Connection header received
0925 22:57:08.984427 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:08.984431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:08.984584 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:08.984757 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:57:08.985448 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.986947 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.989203 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:08.990209 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:08.991408 (+ 1199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:08.991728 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:08.992627 (+ 899us) server_negotiation.cc:299] Negotiation successful
0925 22:57:08.992757 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":937,"thread_start_us":113,"threads_started":1}
I20260925 22:57:08.994223 2646 tablet_service.cc:1467] Processing CreateTablet for tablet 261e029a545a4b6a80a68d1b3dbfdf86 (DEFAULT_TABLE table=test-table [id=e0b71521c7f64b20908867cf651ca30b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:57:08.995687 2646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 261e029a545a4b6a80a68d1b3dbfdf86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:57:08.996443 2512 tablet_service.cc:1467] Processing CreateTablet for tablet 261e029a545a4b6a80a68d1b3dbfdf86 (DEFAULT_TABLE table=test-table [id=e0b71521c7f64b20908867cf651ca30b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:57:08.996605 2378 tablet_service.cc:1467] Processing CreateTablet for tablet 261e029a545a4b6a80a68d1b3dbfdf86 (DEFAULT_TABLE table=test-table [id=e0b71521c7f64b20908867cf651ca30b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:57:08.997933 2378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 261e029a545a4b6a80a68d1b3dbfdf86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:57:08.997984 2512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 261e029a545a4b6a80a68d1b3dbfdf86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:57:09.005409 2729 tablet_bootstrap.cc:492] T 261e029a545a4b6a80a68d1b3dbfdf86 P 28ef18fda7e441508e538985b7fdc630: Bootstrap starting.
I20260925 22:57:09.008268 2731 tablet_bootstrap.cc:492] T 261e029a545a4b6a80a68d1b3dbfdf86 P 956e81b9add74bad987c4aa1d19ab4e9: Bootstrap starting.
I20260925 22:57:09.008800 2729 tablet_bootstrap.cc:654] T 261e029a545a4b6a80a68d1b3dbfdf86 P 28ef18fda7e441508e538985b7fdc630: Neither blocks nor log segments found. Creating new log.
I20260925 22:57:09.009590 2730 tablet_bootstrap.cc:492] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff: Bootstrap starting.
I20260925 22:57:09.011548 2729 log.cc:824] T 261e029a545a4b6a80a68d1b3dbfdf86 P 28ef18fda7e441508e538985b7fdc630: Log is configured to *not* fsync() on all Append() calls
I20260925 22:57:09.011878 2731 tablet_bootstrap.cc:654] T 261e029a545a4b6a80a68d1b3dbfdf86 P 956e81b9add74bad987c4aa1d19ab4e9: Neither blocks nor log segments found. Creating new log.
I20260925 22:57:09.012707 2730 tablet_bootstrap.cc:654] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff: Neither blocks nor log segments found. Creating new log.
I20260925 22:57:09.013181 2731 log.cc:824] T 261e029a545a4b6a80a68d1b3dbfdf86 P 956e81b9add74bad987c4aa1d19ab4e9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:57:09.013701 2730 log.cc:824] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff: Log is configured to *not* fsync() on all Append() calls
I20260925 22:57:09.015470 2729 tablet_bootstrap.cc:492] T 261e029a545a4b6a80a68d1b3dbfdf86 P 28ef18fda7e441508e538985b7fdc630: No bootstrap required, opened a new log
I20260925 22:57:09.016194 2730 tablet_bootstrap.cc:492] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff: No bootstrap required, opened a new log
I20260925 22:57:09.016202 2729 ts_tablet_manager.cc:1397] T 261e029a545a4b6a80a68d1b3dbfdf86 P 28ef18fda7e441508e538985b7fdc630: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260925 22:57:09.016453 2730 ts_tablet_manager.cc:1397] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:57:09.018328 2731 tablet_bootstrap.cc:492] T 261e029a545a4b6a80a68d1b3dbfdf86 P 956e81b9add74bad987c4aa1d19ab4e9: No bootstrap required, opened a new log
I20260925 22:57:09.018702 2731 ts_tablet_manager.cc:1397] T 261e029a545a4b6a80a68d1b3dbfdf86 P 956e81b9add74bad987c4aa1d19ab4e9: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260925 22:57:09.021580 2729 raft_consensus.cc:348] T 261e029a545a4b6a80a68d1b3dbfdf86 P 28ef18fda7e441508e538985b7fdc630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:09.021981 2729 raft_consensus.cc:374] T 261e029a545a4b6a80a68d1b3dbfdf86 P 28ef18fda7e441508e538985b7fdc630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:57:09.021826 2730 raft_consensus.cc:348] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:09.022121 2729 raft_consensus.cc:729] T 261e029a545a4b6a80a68d1b3dbfdf86 P 28ef18fda7e441508e538985b7fdc630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28ef18fda7e441508e538985b7fdc630, State: Initialized, Role: FOLLOWER
I20260925 22:57:09.022166 2730 raft_consensus.cc:374] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:57:09.022228 2730 raft_consensus.cc:729] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f07077db5dbd405d968bf7b0da05e2ff, State: Initialized, Role: FOLLOWER
I20260925 22:57:09.022742 2729 consensus_queue.cc:260] T 261e029a545a4b6a80a68d1b3dbfdf86 P 28ef18fda7e441508e538985b7fdc630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:09.023205 2730 consensus_queue.cc:260] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:09.023304 2731 raft_consensus.cc:348] T 261e029a545a4b6a80a68d1b3dbfdf86 P 956e81b9add74bad987c4aa1d19ab4e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:09.023620 2731 raft_consensus.cc:374] T 261e029a545a4b6a80a68d1b3dbfdf86 P 956e81b9add74bad987c4aa1d19ab4e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:57:09.023679 2731 raft_consensus.cc:729] T 261e029a545a4b6a80a68d1b3dbfdf86 P 956e81b9add74bad987c4aa1d19ab4e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 956e81b9add74bad987c4aa1d19ab4e9, State: Initialized, Role: FOLLOWER
I20260925 22:57:09.024127 2729 ts_tablet_manager.cc:1428] T 261e029a545a4b6a80a68d1b3dbfdf86 P 28ef18fda7e441508e538985b7fdc630: Time spent starting tablet: real 0.008s user 0.010s sys 0.000s
I20260925 22:57:09.024163 2731 consensus_queue.cc:260] T 261e029a545a4b6a80a68d1b3dbfdf86 P 956e81b9add74bad987c4aa1d19ab4e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:09.024417 2444 heartbeater.cc:503] Master 127.20.114.254:43183 was elected leader, sending a full tablet report...
I20260925 22:57:09.024542 2712 heartbeater.cc:503] Master 127.20.114.254:43183 was elected leader, sending a full tablet report...
I20260925 22:57:09.024603 2730 ts_tablet_manager.cc:1428] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260925 22:57:09.025563 2731 ts_tablet_manager.cc:1428] T 261e029a545a4b6a80a68d1b3dbfdf86 P 956e81b9add74bad987c4aa1d19ab4e9: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 22:57:09.025991 2578 heartbeater.cc:503] Master 127.20.114.254:43183 was elected leader, sending a full tablet report...
W20260925 22:57:09.039738 2445 tablet.cc:2369] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:57:09.098238 2579 tablet.cc:2369] T 261e029a545a4b6a80a68d1b3dbfdf86 P 956e81b9add74bad987c4aa1d19ab4e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:57:09.144490 2736 raft_consensus.cc:482] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:57:09.144750 2736 raft_consensus.cc:504] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:09.146263 2736 leader_election.cc:288] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 956e81b9add74bad987c4aa1d19ab4e9 (127.20.114.194:45971), 28ef18fda7e441508e538985b7fdc630 (127.20.114.195:43563)
I20260925 22:57:09.151535 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:09.146498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:45971 (local address 127.20.114.193:49729)
0925 22:57:09.146949 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:09.146967 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:09.147159 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:09.147560 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:09.147563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:09.147579 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:57:09.147766 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:09.147775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:09.148676 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:09.148680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:09.150174 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:09.150190 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:09.151021 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:09.151028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:09.151148 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:09.151176 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 22:57:09.151258 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 22:57:09.151330 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":206,"thread_start_us":147,"threads_started":1}
I20260925 22:57:09.151571 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:09.146634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:49729 (local address 127.20.114.194:45971)
0925 22:57:09.146798 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:09.146803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:09.147064 (+ 261us) server_negotiation.cc:407] Connection header received
0925 22:57:09.147294 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:09.147298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:09.147357 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:09.147408 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:57:09.147907 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:09.148548 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:09.150367 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:09.150847 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:09.150976 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:09.151024 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:09.151304 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 22:57:09.151402 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:57:09.151767 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:09.147113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:41367 (local address 127.20.114.195:43563)
0925 22:57:09.147267 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:09.147273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:09.147294 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:57:09.147471 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:09.147474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:09.147540 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:09.147637 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:57:09.148106 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:09.148759 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:09.150192 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:09.150972 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:09.151118 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:09.151170 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:09.151530 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 22:57:09.151620 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 22:57:09.151770 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:09.146631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:43563 (local address 127.20.114.193:41367)
0925 22:57:09.147210 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:09.147227 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:09.147352 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:09.147647 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:09.147650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:09.147667 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:57:09.147845 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:09.147855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:09.148979 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:09.148989 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:09.150014 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:09.150025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:09.151251 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:09.151257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:09.151373 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:09.151397 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:57:09.151487 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 22:57:09.151558 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":30,"thread_start_us":116,"threads_started":1}
I20260925 22:57:09.153190 2531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "261e029a545a4b6a80a68d1b3dbfdf86" candidate_uuid: "f07077db5dbd405d968bf7b0da05e2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "956e81b9add74bad987c4aa1d19ab4e9" is_pre_election: true
I20260925 22:57:09.153173 2666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "261e029a545a4b6a80a68d1b3dbfdf86" candidate_uuid: "f07077db5dbd405d968bf7b0da05e2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28ef18fda7e441508e538985b7fdc630" is_pre_election: true
I20260925 22:57:09.153587 2531 raft_consensus.cc:2454] T 261e029a545a4b6a80a68d1b3dbfdf86 P 956e81b9add74bad987c4aa1d19ab4e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f07077db5dbd405d968bf7b0da05e2ff in term 0.
I20260925 22:57:09.153811 2666 raft_consensus.cc:2454] T 261e029a545a4b6a80a68d1b3dbfdf86 P 28ef18fda7e441508e538985b7fdc630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f07077db5dbd405d968bf7b0da05e2ff in term 0.
I20260925 22:57:09.154369 2334 leader_election.cc:302] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28ef18fda7e441508e538985b7fdc630, 956e81b9add74bad987c4aa1d19ab4e9, f07077db5dbd405d968bf7b0da05e2ff; no voters:
I20260925 22:57:09.154704 2736 raft_consensus.cc:2781] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:57:09.154804 2736 raft_consensus.cc:482] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:57:09.154851 2736 raft_consensus.cc:3037] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:57:09.156046 2736 raft_consensus.cc:504] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:09.156504 2736 leader_election.cc:288] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 1 election: Requested vote from peers 956e81b9add74bad987c4aa1d19ab4e9 (127.20.114.194:45971), 28ef18fda7e441508e538985b7fdc630 (127.20.114.195:43563)
I20260925 22:57:09.157229 2531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "261e029a545a4b6a80a68d1b3dbfdf86" candidate_uuid: "f07077db5dbd405d968bf7b0da05e2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "956e81b9add74bad987c4aa1d19ab4e9"
I20260925 22:57:09.157210 2666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "261e029a545a4b6a80a68d1b3dbfdf86" candidate_uuid: "f07077db5dbd405d968bf7b0da05e2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28ef18fda7e441508e538985b7fdc630"
I20260925 22:57:09.157418 2531 raft_consensus.cc:3037] T 261e029a545a4b6a80a68d1b3dbfdf86 P 956e81b9add74bad987c4aa1d19ab4e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:57:09.157444 2666 raft_consensus.cc:3037] T 261e029a545a4b6a80a68d1b3dbfdf86 P 28ef18fda7e441508e538985b7fdc630 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:57:09.158617 2531 raft_consensus.cc:2454] T 261e029a545a4b6a80a68d1b3dbfdf86 P 956e81b9add74bad987c4aa1d19ab4e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f07077db5dbd405d968bf7b0da05e2ff in term 1.
I20260925 22:57:09.158950 2666 raft_consensus.cc:2454] T 261e029a545a4b6a80a68d1b3dbfdf86 P 28ef18fda7e441508e538985b7fdc630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f07077db5dbd405d968bf7b0da05e2ff in term 1.
I20260925 22:57:09.159089 2334 leader_election.cc:302] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 956e81b9add74bad987c4aa1d19ab4e9, f07077db5dbd405d968bf7b0da05e2ff; no voters:
I20260925 22:57:09.159340 2736 raft_consensus.cc:2781] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:57:09.159775 2736 raft_consensus.cc:686] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [term 1 LEADER]: Becoming Leader. State: Replica: f07077db5dbd405d968bf7b0da05e2ff, State: Running, Role: LEADER
I20260925 22:57:09.160293 2736 consensus_queue.cc:237] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
W20260925 22:57:09.163127 2713 tablet.cc:2369] T 261e029a545a4b6a80a68d1b3dbfdf86 P 28ef18fda7e441508e538985b7fdc630: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:57:09.164047 2253 catalog_manager.cc:5495] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff reported cstate change: term changed from 0 to 1, leader changed from <none> to f07077db5dbd405d968bf7b0da05e2ff (127.20.114.193). New cstate: current_term: 1 leader_uuid: "f07077db5dbd405d968bf7b0da05e2ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } health_report { overall_health: HEALTHY } } }
I20260925 22:57:09.196779 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:09.192384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45458 (local address 127.20.114.193:39053)
0925 22:57:09.192565 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:09.192573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:09.192592 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:57:09.192692 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:09.192695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:09.192753 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:09.192917 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:57:09.193224 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:09.193789 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:09.194790 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:09.195047 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:09.195179 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:09.195749 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:57:09.196245 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:57:09.196396 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:09.196445 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 22:57:09.196524 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 22:57:09.204023 2666 raft_consensus.cc:1261] T 261e029a545a4b6a80a68d1b3dbfdf86 P 28ef18fda7e441508e538985b7fdc630 [term 1 FOLLOWER]: Refusing update from remote peer f07077db5dbd405d968bf7b0da05e2ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:57:09.204023 2531 raft_consensus.cc:1261] T 261e029a545a4b6a80a68d1b3dbfdf86 P 956e81b9add74bad987c4aa1d19ab4e9 [term 1 FOLLOWER]: Refusing update from remote peer f07077db5dbd405d968bf7b0da05e2ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:57:09.204944 2740 consensus_queue.cc:1035] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:57:09.205190 2736 consensus_queue.cc:1035] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:57:09.214167 2744 mvcc.cc:205] Tried to move back new op lower bound from 7333384311610388480 to 7333384311443849216. Current Snapshot: MvccSnapshot[applied={T|T < 7333384311610388480}]
I20260925 22:57:09.224292 2251 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:57:09.224555 2251 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:57:09.227991 2251 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2: Sending DeleteTablet for 3 replicas of tablet 261e029a545a4b6a80a68d1b3dbfdf86
I20260925 22:57:09.229275 2512 tablet_service.cc:1514] Processing DeleteTablet for tablet 261e029a545a4b6a80a68d1b3dbfdf86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:57:09 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:51368
I20260925 22:57:09.229238 2646 tablet_service.cc:1514] Processing DeleteTablet for tablet 261e029a545a4b6a80a68d1b3dbfdf86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:57:09 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:39444
I20260925 22:57:09.229228 2378 tablet_service.cc:1514] Processing DeleteTablet for tablet 261e029a545a4b6a80a68d1b3dbfdf86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:57:09 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:45448
I20260925 22:57:09.230021 2753 tablet_replica.cc:330] T 261e029a545a4b6a80a68d1b3dbfdf86 P 28ef18fda7e441508e538985b7fdc630: stopping tablet replica
I20260925 22:57:09.230235 2754 tablet_replica.cc:330] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff: stopping tablet replica
I20260925 22:57:09.230381 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 2315
I20260925 22:57:09.230578 2753 raft_consensus.cc:2229] T 261e029a545a4b6a80a68d1b3dbfdf86 P 28ef18fda7e441508e538985b7fdc630 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:57:09.231117 2753 raft_consensus.cc:2258] T 261e029a545a4b6a80a68d1b3dbfdf86 P 28ef18fda7e441508e538985b7fdc630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:57:09.231346 2752 tablet_replica.cc:330] T 261e029a545a4b6a80a68d1b3dbfdf86 P 956e81b9add74bad987c4aa1d19ab4e9: stopping tablet replica
I20260925 22:57:09.232486 2752 raft_consensus.cc:2229] T 261e029a545a4b6a80a68d1b3dbfdf86 P 956e81b9add74bad987c4aa1d19ab4e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:57:09.232949 2752 raft_consensus.cc:2258] T 261e029a545a4b6a80a68d1b3dbfdf86 P 956e81b9add74bad987c4aa1d19ab4e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:57:09.234189 2753 ts_tablet_manager.cc:1905] T 261e029a545a4b6a80a68d1b3dbfdf86 P 28ef18fda7e441508e538985b7fdc630: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:57:09.234915 2752 ts_tablet_manager.cc:1905] T 261e029a545a4b6a80a68d1b3dbfdf86 P 956e81b9add74bad987c4aa1d19ab4e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:57:09.238627 2753 ts_tablet_manager.cc:1918] T 261e029a545a4b6a80a68d1b3dbfdf86 P 28ef18fda7e441508e538985b7fdc630: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:57:09.238786 2753 log.cc:1196] T 261e029a545a4b6a80a68d1b3dbfdf86 P 28ef18fda7e441508e538985b7fdc630: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/261e029a545a4b6a80a68d1b3dbfdf86
I20260925 22:57:09.239471 2753 ts_tablet_manager.cc:1939] T 261e029a545a4b6a80a68d1b3dbfdf86 P 28ef18fda7e441508e538985b7fdc630: Deleting consensus metadata
I20260925 22:57:09.239514 2752 ts_tablet_manager.cc:1918] T 261e029a545a4b6a80a68d1b3dbfdf86 P 956e81b9add74bad987c4aa1d19ab4e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260925 22:57:09.239605 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.114.193:39053 (error 108)
I20260925 22:57:09.239691 2752 log.cc:1196] T 261e029a545a4b6a80a68d1b3dbfdf86 P 956e81b9add74bad987c4aa1d19ab4e9: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/261e029a545a4b6a80a68d1b3dbfdf86
I20260925 22:57:09.239990 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 2449
I20260925 22:57:09.240036 2752 ts_tablet_manager.cc:1939] T 261e029a545a4b6a80a68d1b3dbfdf86 P 956e81b9add74bad987c4aa1d19ab4e9: Deleting consensus metadata
I20260925 22:57:09.241365 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:09.240887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:39053 (local address 127.0.0.1:45472)
0925 22:57:09.241133 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:09.241250 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.193:39053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 22:57:09.241919 2237 catalog_manager.cc:4584] TS f07077db5dbd405d968bf7b0da05e2ff (127.20.114.193:39053): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 261e029a545a4b6a80a68d1b3dbfdf86: Network error: Client connection negotiation failed: client connection to 127.20.114.193:39053: connect: Connection refused (error 111)
I20260925 22:57:09.242250 2239 catalog_manager.cc:4841] TS 28ef18fda7e441508e538985b7fdc630 (127.20.114.195:43563): tablet 261e029a545a4b6a80a68d1b3dbfdf86 (table test-table [id=e0b71521c7f64b20908867cf651ca30b]) successfully deleted
I20260925 22:57:09.247642 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 2583
I20260925 22:57:09.249251 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:09.248017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:45971 (local address 127.0.0.1:51378)
0925 22:57:09.248814 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:09.249037 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.194:45971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":666}
I20260925 22:57:09.255538 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 2221
I20260925 22:57:09.264580 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:43183
--webserver_interface=127.20.114.254
--webserver_port=44623
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:43183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:57:09.412070 2756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:09.412412 2756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:09.412458 2756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:09.416975 2756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:57:09.417063 2756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:09.417083 2756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:09.417104 2756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:57:09.417119 2756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:57:09.423475 2756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:43183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:43183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=44623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:09.424892 2756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:09.426143 2756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:57:09.443512 2763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:09.443548 2765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:09.443548 2762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:09.444540 2756 server_base.cc:1034] running on GCE node
I20260925 22:57:09.445393 2756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:09.447278 2756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:09.448616 2756 hybrid_clock.cc:648] HybridClock initialized: now 1790377029448560 us; error 69 us; skew 500 ppm
Sep 25 22:57:09 dist-test-slave-x1lh krb5kdc[2208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377029, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:09.460016 2756 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:57:09.461796 2756 webserver.cc:460] Webserver started at http://127.20.114.254:44623/ using document root <none> and password file <none>
I20260925 22:57:09.462596 2756 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:09.462709 2756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:09.468837 2756 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:57:09.472519 2772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:09.474493 2756 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260925 22:57:09.474711 2756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "d3c132d8718d473c94d4fe821ec671f2"
format_stamp: "Formatted at 2026-09-25 22:57:07 on dist-test-slave-x1lh"
server_key: "66b48594ac034fb968b6c47da26d1ed8"
server_key_iv: "1609fa952a5439cb8ceb8caf2f9a099e"
server_key_version: "encryptionkey@0"
I20260925 22:57:09.475464 2756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:09.489588 2756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:09.495224 2756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:09.495641 2756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:09.506628 2756 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:43183
I20260925 22:57:09.507176 2824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:43183 every 8 connection(s)
I20260925 22:57:09.507993 2756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:57:09.515388 2825 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:57:09.515669 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 2756
I20260925 22:57:09.516222 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:39053
--local_ip_for_outbound_sockets=127.20.114.193
--tserver_master_addrs=127.20.114.254:43183
--webserver_port=41425
--webserver_interface=127.20.114.193
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:57:09.518862 2825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2: Bootstrap starting.
I20260925 22:57:09.542665 2825 log.cc:824] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2: Log is configured to *not* fsync() on all Append() calls
I20260925 22:57:09.556794 2825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:57:09.557381 2825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2: Bootstrap complete.
I20260925 22:57:09.562850 2825 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c132d8718d473c94d4fe821ec671f2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43183 } }
I20260925 22:57:09.563374 2825 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3c132d8718d473c94d4fe821ec671f2, State: Initialized, Role: FOLLOWER
I20260925 22:57:09.564188 2825 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c132d8718d473c94d4fe821ec671f2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43183 } }
I20260925 22:57:09.564363 2825 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:57:09.564458 2825 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:57:09.564612 2825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:57:09.567158 2825 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c132d8718d473c94d4fe821ec671f2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43183 } }
I20260925 22:57:09.567632 2825 leader_election.cc:302] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3c132d8718d473c94d4fe821ec671f2; no voters:
I20260925 22:57:09.568171 2825 leader_election.cc:288] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:57:09.568280 2829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:57:09.568557 2829 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 2 LEADER]: Becoming Leader. State: Replica: d3c132d8718d473c94d4fe821ec671f2, State: Running, Role: LEADER
I20260925 22:57:09.568938 2829 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c132d8718d473c94d4fe821ec671f2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43183 } }
I20260925 22:57:09.569784 2825 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:57:09.570894 2830 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3c132d8718d473c94d4fe821ec671f2. Latest consensus state: current_term: 2 leader_uuid: "d3c132d8718d473c94d4fe821ec671f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c132d8718d473c94d4fe821ec671f2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43183 } } }
I20260925 22:57:09.570956 2831 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d3c132d8718d473c94d4fe821ec671f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c132d8718d473c94d4fe821ec671f2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43183 } } }
I20260925 22:57:09.571049 2830 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:57:09.571082 2831 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:57:09.571735 2838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:57:09.578313 2838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:57:09.579412 2838 catalog_manager.cc:1259] Loaded cluster ID: 3512b03610ba46fab4de6dc9b45d6f14
I20260925 22:57:09.579514 2838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:57:09.581761 2838 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 22:57:09.586952 2838 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 22:57:09.594290 2838 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:57:09.595268 2838 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2: Loaded TSK: 0
I20260925 22:57:09.597055 2838 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:57:09.672945 2827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:09.673267 2827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:09.673347 2827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:09.679692 2827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:09.679782 2827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:09.679883 2827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:57:09.687750 2827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:39053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=41425
--tserver_master_addrs=127.20.114.254:43183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:09.689271 2827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:09.690917 2827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:57:09.704344 2853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:09.704487 2827 server_base.cc:1034] running on GCE node
W20260925 22:57:09.704346 2855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:09.704344 2852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:09.705446 2827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:09.706277 2827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:09.707505 2827 hybrid_clock.cc:648] HybridClock initialized: now 1790377029707482 us; error 45 us; skew 500 ppm
Sep 25 22:57:09 dist-test-slave-x1lh krb5kdc[2208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377029, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:09.710873 2827 init.cc:377] Logged in from keytab as kudu/127.20.114.193@KRBTEST.COM (short username kudu)
I20260925 22:57:09.712062 2827 webserver.cc:460] Webserver started at http://127.20.114.193:41425/ using document root <none> and password file <none>
I20260925 22:57:09.712721 2827 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:09.712852 2827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:09.718200 2827 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260925 22:57:09.721477 2862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:09.723584 2827 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:57:09.723742 2827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "f07077db5dbd405d968bf7b0da05e2ff"
format_stamp: "Formatted at 2026-09-25 22:57:08 on dist-test-slave-x1lh"
server_key: "091208c7f52b0b69e7a6964c3e9cb66d"
server_key_iv: "3cf1ccd24414325ba917f4a7ea395ea1"
server_key_version: "encryptionkey@0"
I20260925 22:57:09.724469 2827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:09.740437 2827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:09.744699 2827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:09.745020 2827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:09.745775 2827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:57:09.746964 2869 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:57:09.753657 2827 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:57:09.753773 2827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260925 22:57:09.753856 2827 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:57:09.755942 2827 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:57:09.756049 2827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 22:57:09.756376 2869 tablet_bootstrap.cc:492] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff: Bootstrap starting.
I20260925 22:57:09.768711 2827 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:39053
I20260925 22:57:09.768817 2976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:39053 every 8 connection(s)
I20260925 22:57:09.770349 2827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:57:09.778585 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 2827
I20260925 22:57:09.778903 2869 log.cc:824] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff: Log is configured to *not* fsync() on all Append() calls
I20260925 22:57:09.779394 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:45971
--local_ip_for_outbound_sockets=127.20.114.194
--tserver_master_addrs=127.20.114.254:43183
--webserver_port=39915
--webserver_interface=127.20.114.194
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:57:09 dist-test-slave-x1lh krb5kdc[2208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377029, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:57:09.791352 2869 tablet_bootstrap.cc:492] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:57:09.792196 2869 tablet_bootstrap.cc:492] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff: Bootstrap complete.
I20260925 22:57:09.793943 2869 ts_tablet_manager.cc:1397] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff: Time spent bootstrapping tablet: real 0.038s user 0.025s sys 0.011s
I20260925 22:57:09.795341 2980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:09.772769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:46821 (local address 127.20.114.254:43183)
0925 22:57:09.773558 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:09.773586 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:09.773647 (+ 61us) server_negotiation.cc:407] Connection header received
0925 22:57:09.774943 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:09.774969 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:09.775509 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:09.775990 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:09.777837 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:09.779465 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:09.781943 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:09.782882 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:09.783229 (+ 347us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:09.787423 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:09.787429 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:09.787445 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:09.787481 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:09.790188 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:09.791576 (+ 1388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:09.791583 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:09.791588 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:09.791860 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:09.792588 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:09.792592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:09.792594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:09.793101 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:09.793265 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:09.793871 (+ 606us) server_negotiation.cc:299] Negotiation successful
0925 22:57:09.794265 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":283,"threads_started":1}
I20260925 22:57:09.795339 2979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:09.772772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.193:46821)
0925 22:57:09.773384 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:09.773447 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:09.774331 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:09.776086 (+ 1755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:09.776094 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:09.776806 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:09.777597 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:09.777617 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:09.779875 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:09.779879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:09.781636 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:09.781651 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:09.783210 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:09.783215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:09.783431 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:09.784736 (+ 1305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:09.784757 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:09.787136 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:09.790368 (+ 3232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:09.790375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:09.790390 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:09.791306 (+ 916us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:09.792228 (+ 922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:09.792232 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:09.792234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:09.792455 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:09.793229 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:09.793236 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:09.793514 (+ 278us) client_negotiation.cc:769] Sending connection context
0925 22:57:09.793710 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 22:57:09.794138 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":181,"threads_started":1}
I20260925 22:57:09.798455 2977 heartbeater.cc:347] Connected to a master server at 127.20.114.254:43183
I20260925 22:57:09.797861 2869 raft_consensus.cc:348] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:09.798626 2869 raft_consensus.cc:729] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f07077db5dbd405d968bf7b0da05e2ff, State: Initialized, Role: FOLLOWER
I20260925 22:57:09.798725 2977 heartbeater.cc:464] Registering TS with master...
I20260925 22:57:09.799192 2869 consensus_queue.cc:260] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:09.799285 2977 heartbeater.cc:511] Master 127.20.114.254:43183 requested a full tablet report, sending...
I20260925 22:57:09.800346 2869 ts_tablet_manager.cc:1428] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260925 22:57:09.802119 2787 ts_manager.cc:194] Registered new tserver with Master: f07077db5dbd405d968bf7b0da05e2ff (127.20.114.193:39053)
I20260925 22:57:09.807447 2787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:46821
I20260925 22:57:09.811213 2977 heartbeater.cc:503] Master 127.20.114.254:43183 was elected leader, sending a full tablet report...
Sep 25 22:57:09 dist-test-slave-x1lh krb5kdc[2208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377029, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for kudu/127.20.114.193@KRBTEST.COM
I20260925 22:57:09.820175 2984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:09.806282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:39053 (local address 127.0.0.1:45488)
0925 22:57:09.807214 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:09.807273 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:09.807575 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:09.808712 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:09.808718 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:09.809173 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:09.809290 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:09.809297 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:09.810786 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:09.810793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:09.811607 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:09.811620 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:09.812452 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:09.812458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:09.812604 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:09.813332 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:09.813355 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:09.814911 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:09.817599 (+ 2688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:09.817629 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:09.817634 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:09.817921 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:09.818358 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:09.818360 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:09.818362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:09.818699 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:09.819460 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:09.819466 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:09.819603 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 22:57:09.819744 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 22:57:09.820025 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":259,"threads_started":1}
I20260925 22:57:09.820367 2985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:09.806350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45488 (local address 127.20.114.193:39053)
0925 22:57:09.806784 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:09.806799 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:09.807389 (+ 590us) server_negotiation.cc:407] Connection header received
0925 22:57:09.808110 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:09.808130 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:09.808401 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:09.808569 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:09.809422 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:09.810394 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:09.811785 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:09.812303 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:09.812426 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:09.815101 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:09.815116 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:09.815132 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:09.815163 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:09.817333 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:09.818081 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:09.818090 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:09.818091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:09.818203 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:09.818878 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:09.818882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:09.818884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:09.819252 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:09.819399 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:09.819879 (+ 480us) server_negotiation.cc:299] Negotiation successful
0925 22:57:09.820192 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":163,"threads_started":1}
I20260925 22:57:09.821805 2911 tablet_service.cc:1514] Processing DeleteTablet for tablet 261e029a545a4b6a80a68d1b3dbfdf86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:57:09 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:45488
I20260925 22:57:09.823293 2986 tablet_replica.cc:330] stopping tablet replica
I20260925 22:57:09.823658 2986 raft_consensus.cc:2229] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:57:09.823936 2986 raft_consensus.cc:2258] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:57:09.825251 2986 ts_tablet_manager.cc:1905] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:57:09.830471 2986 ts_tablet_manager.cc:1918] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:57:09.830581 2986 log.cc:1196] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/wal/wals/261e029a545a4b6a80a68d1b3dbfdf86
I20260925 22:57:09.830878 2986 ts_tablet_manager.cc:1939] T 261e029a545a4b6a80a68d1b3dbfdf86 P f07077db5dbd405d968bf7b0da05e2ff: Deleting consensus metadata
I20260925 22:57:09.833479 2773 catalog_manager.cc:4841] TS f07077db5dbd405d968bf7b0da05e2ff (127.20.114.193:39053): tablet 261e029a545a4b6a80a68d1b3dbfdf86 (table test-table [id=e0b71521c7f64b20908867cf651ca30b]) successfully deleted
W20260925 22:57:09.929265 2982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:09.929687 2982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:09.929781 2982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:09.934569 2982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:09.934703 2982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:09.934810 2982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:57:09.940763 2982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:45971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=39915
--tserver_master_addrs=127.20.114.254:43183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:09.942049 2982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:09.943146 2982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:57:09.955638 2994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:09.955663 2992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:09.955673 2991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:09.956319 2982 server_base.cc:1034] running on GCE node
I20260925 22:57:09.956988 2982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:09.958329 2982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:09.959542 2982 hybrid_clock.cc:648] HybridClock initialized: now 1790377029959502 us; error 54 us; skew 500 ppm
Sep 25 22:57:09 dist-test-slave-x1lh krb5kdc[2208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377029, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:09.965353 2982 init.cc:377] Logged in from keytab as kudu/127.20.114.194@KRBTEST.COM (short username kudu)
I20260925 22:57:09.967397 2982 webserver.cc:460] Webserver started at http://127.20.114.194:39915/ using document root <none> and password file <none>
I20260925 22:57:09.968326 2982 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:09.968464 2982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:09.974766 2982 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:57:09.977870 3001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:09.980144 2982 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:57:09.980355 2982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "956e81b9add74bad987c4aa1d19ab4e9"
format_stamp: "Formatted at 2026-09-25 22:57:08 on dist-test-slave-x1lh"
server_key: "0fe5227eaa09142190666b24a221c256"
server_key_iv: "16dfdc12d09536d911f45728e4bfd5f2"
server_key_version: "encryptionkey@0"
I20260925 22:57:09.981647 2982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:10.004185 2982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:10.009004 2982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:10.009289 2982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:10.010174 2982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:57:10.011400 2982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:57:10.011521 2982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:10.011600 2982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:57:10.011653 2982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:10.023810 2982 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:45971
I20260925 22:57:10.023937 3114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:45971 every 8 connection(s)
I20260925 22:57:10.025101 2982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:57:10.032552 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 2982
I20260925 22:57:10.033093 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:43563
--local_ip_for_outbound_sockets=127.20.114.195
--tserver_master_addrs=127.20.114.254:43183
--webserver_port=34645
--webserver_interface=127.20.114.195
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:57:10 dist-test-slave-x1lh krb5kdc[2208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377029, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:57:10.044092 2980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:10.027309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:42955 (local address 127.20.114.254:43183)
0925 22:57:10.027487 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:10.027491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:10.028845 (+ 1354us) server_negotiation.cc:407] Connection header received
0925 22:57:10.030694 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:10.030698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:10.030775 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:10.030894 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:10.032361 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.033069 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.034440 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.034710 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.034838 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:10.038424 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:10.038427 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:10.038429 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:10.038462 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:10.040154 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:10.041648 (+ 1494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:10.041653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:10.041666 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:10.041881 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:10.042619 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:10.042622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:10.042626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:10.042858 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:10.043096 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:10.043739 (+ 643us) server_negotiation.cc:299] Negotiation successful
0925 22:57:10.043909 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 22:57:10.045387 3117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:10.027928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.194:42955)
0925 22:57:10.028508 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:10.028571 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:10.030362 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:10.031123 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:10.031132 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:10.031571 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:10.032157 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:10.032176 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.033239 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.033242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:10.034266 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:10.034279 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.034914 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.034918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:10.035122 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:10.036287 (+ 1165us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:10.036307 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:10.038165 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:10.040333 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:10.040342 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:10.040366 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:10.041333 (+ 967us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:10.042262 (+ 929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:10.042268 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:10.042270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:10.042493 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:10.043089 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:10.043097 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:10.043432 (+ 335us) client_negotiation.cc:769] Sending connection context
0925 22:57:10.043670 (+ 238us) client_negotiation.cc:240] Negotiation successful
0925 22:57:10.044096 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":143,"threads_started":1}
I20260925 22:57:10.046999 3115 heartbeater.cc:347] Connected to a master server at 127.20.114.254:43183
I20260925 22:57:10.047303 3115 heartbeater.cc:464] Registering TS with master...
I20260925 22:57:10.048055 3115 heartbeater.cc:511] Master 127.20.114.254:43183 requested a full tablet report, sending...
I20260925 22:57:10.049819 2787 ts_manager.cc:194] Registered new tserver with Master: 956e81b9add74bad987c4aa1d19ab4e9 (127.20.114.194:45971)
I20260925 22:57:10.050762 2787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:42955
W20260925 22:57:10.177132 3118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:10.177497 3118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:10.177578 3118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:10.183354 3118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:10.183662 3118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:10.183833 3118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:57:10.192233 3118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:43563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=34645
--tserver_master_addrs=127.20.114.254:43183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:10.193542 3118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:10.195119 3118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:57:10.207836 3124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:10.208089 3118 server_base.cc:1034] running on GCE node
W20260925 22:57:10.207836 3123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:10.207836 3126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:10.209056 3118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:10.209810 3118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:10.211036 3118 hybrid_clock.cc:648] HybridClock initialized: now 1790377030211017 us; error 36 us; skew 500 ppm
Sep 25 22:57:10 dist-test-slave-x1lh krb5kdc[2208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377030, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:10.215313 3118 init.cc:377] Logged in from keytab as kudu/127.20.114.195@KRBTEST.COM (short username kudu)
I20260925 22:57:10.217224 3118 webserver.cc:460] Webserver started at http://127.20.114.195:34645/ using document root <none> and password file <none>
I20260925 22:57:10.218149 3118 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:10.218315 3118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:10.223515 3118 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 22:57:10.226679 3133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:10.228708 3118 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:57:10.228963 3118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "28ef18fda7e441508e538985b7fdc630"
format_stamp: "Formatted at 2026-09-25 22:57:08 on dist-test-slave-x1lh"
server_key: "2f59199e402207f35ca0849c8718ebdc"
server_key_iv: "9805d192c6518ec0a25a047058eb83b6"
server_key_version: "encryptionkey@0"
I20260925 22:57:10.229692 3118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:10.258097 3118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:10.263319 3118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:10.263592 3118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:10.264314 3118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:57:10.265309 3118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:57:10.265388 3118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:10.265475 3118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:57:10.265519 3118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:10.279824 3118 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:43563
I20260925 22:57:10.279845 3246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:43563 every 8 connection(s)
I20260925 22:57:10.281176 3118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:57:10.284224 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 3118
Sep 25 22:57:10 dist-test-slave-x1lh krb5kdc[2208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377030, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:57:10.297509 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:10.285284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52934 (local address 127.20.114.254:43183)
0925 22:57:10.285658 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:10.285665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:10.286055 (+ 390us) server_negotiation.cc:407] Connection header received
0925 22:57:10.286347 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:10.286350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:10.286401 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:10.286511 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:10.287784 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.288383 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.289809 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.290221 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.290383 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:10.292499 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:10.292502 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:10.292504 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:10.292536 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:10.295059 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:10.295829 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:10.295833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:10.295834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:10.295932 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:10.296312 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:10.296316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:10.296317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:10.296546 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:10.296981 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:10.297160 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 22:57:10.297306 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":287,"thread_start_us":100,"threads_started":1}
I20260925 22:57:10.304037 2980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:10.283733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:48741 (local address 127.20.114.254:43183)
0925 22:57:10.283946 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:10.283951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:10.285781 (+ 1830us) server_negotiation.cc:407] Connection header received
0925 22:57:10.288225 (+ 2444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:10.288228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:10.288291 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:10.288539 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:10.290381 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.290987 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.293340 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.293824 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.294012 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:10.297754 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:10.297758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:10.297761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:10.297787 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:10.299981 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:10.300694 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:10.300698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:10.300699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:10.300842 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:10.301476 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:10.301482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:10.301483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:10.302108 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:10.302329 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:10.303604 (+ 1275us) server_negotiation.cc:299] Negotiation successful
0925 22:57:10.303839 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":83}
I20260925 22:57:10.305086 3249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:10.284377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.195:48741)
0925 22:57:10.285443 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:10.285618 (+ 175us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:10.287924 (+ 2306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:10.288774 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:10.288783 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:10.289291 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:10.290199 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:10.290219 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.291218 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.291223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:10.292796 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:10.292809 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.294244 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.294250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:10.294482 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:10.295103 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:10.295121 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:10.297523 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:10.300202 (+ 2679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:10.300209 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:10.300225 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:10.300564 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:10.301010 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:10.301013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:10.301014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:10.301190 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:10.302366 (+ 1176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:10.302373 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:10.303030 (+ 657us) client_negotiation.cc:769] Sending connection context
0925 22:57:10.303378 (+ 348us) client_negotiation.cc:240] Negotiation successful
0925 22:57:10.303738 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":258,"threads_started":1}
I20260925 22:57:10.306931 3247 heartbeater.cc:347] Connected to a master server at 127.20.114.254:43183
I20260925 22:57:10.307374 3247 heartbeater.cc:464] Registering TS with master...
I20260925 22:57:10.307984 3247 heartbeater.cc:511] Master 127.20.114.254:43183 requested a full tablet report, sending...
I20260925 22:57:10.309808 2787 ts_manager.cc:194] Registered new tserver with Master: 28ef18fda7e441508e538985b7fdc630 (127.20.114.195:43563)
I20260925 22:57:10.310439 2787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:48741
I20260925 22:57:10.312744 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:57:10.318655 2980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:10.313535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.20.114.254:43183)
0925 22:57:10.313754 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:10.313759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:10.313927 (+ 168us) server_negotiation.cc:407] Connection header received
0925 22:57:10.314447 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:10.314452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:10.314829 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:10.315048 (+ 219us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:57:10.315361 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.315907 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.316844 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.317127 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.317250 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:10.317689 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:57:10.318177 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:57:10.318336 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:10.318407 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 22:57:10.318475 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260925 22:57:10.319725 2787 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:52948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:57:10.323956 2787 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:57:10.335695 2911 tablet_service.cc:1467] Processing CreateTablet for tablet 36511de17eed42118a1aec6d5edc25a5 (DEFAULT_TABLE table=test-table [id=2d98669447e549a89c98989962b8ff31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:57:10.336520 2911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36511de17eed42118a1aec6d5edc25a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:57:10.340061 3260 tablet_bootstrap.cc:492] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff: Bootstrap starting.
I20260925 22:57:10.341444 3260 tablet_bootstrap.cc:654] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff: Neither blocks nor log segments found. Creating new log.
I20260925 22:57:10.341987 3256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:10.333986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:43563 (local address 127.0.0.1:39458)
0925 22:57:10.334499 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:10.334516 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:10.334617 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:10.336001 (+ 1384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:10.336004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:10.336121 (+ 117us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:57:10.336279 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:10.336287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.338157 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.338162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:10.339490 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:10.339684 (+ 194us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.341580 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.341584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:10.341684 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:10.341711 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:57:10.341766 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 22:57:10.341828 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":6,"thread_start_us":93,"threads_started":1}
I20260925 22:57:10.342104 3259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:10.334395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39458 (local address 127.20.114.195:43563)
0925 22:57:10.335396 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:10.335411 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:10.335436 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:57:10.335520 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:10.335527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:10.335773 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:10.336430 (+ 657us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:57:10.336636 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.337965 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.340365 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.341446 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.341589 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:10.341748 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:10.341814 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 22:57:10.341924 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":894,"thread_start_us":174,"threads_started":1}
I20260925 22:57:10.342697 3258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:10.334428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51392 (local address 127.20.114.194:45971)
0925 22:57:10.335396 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:10.335411 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:10.335436 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:57:10.335531 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:10.335536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:10.335773 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:10.336316 (+ 543us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:57:10.336765 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.338143 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.340151 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.341181 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.341339 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:10.341497 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:10.341905 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 22:57:10.342129 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":852,"thread_start_us":140,"threads_started":1}
I20260925 22:57:10.342932 3260 tablet_bootstrap.cc:492] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff: No bootstrap required, opened a new log
I20260925 22:57:10.343046 3260 ts_tablet_manager.cc:1397] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:57:10.343652 3257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:10.334195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:45971 (local address 127.0.0.1:51392)
0925 22:57:10.334974 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:10.334988 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:10.335137 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:10.336512 (+ 1375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:10.336515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:10.336532 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:57:10.336626 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:10.336632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.338362 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.338365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:10.339879 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:10.339893 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.341327 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.341331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:10.341423 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:10.341486 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 22:57:10.341708 (+ 222us) client_negotiation.cc:240] Negotiation successful
0925 22:57:10.341816 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":51,"thread_start_us":88,"threads_started":1}
I20260925 22:57:10.343958 3260 raft_consensus.cc:348] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:10.344097 3260 raft_consensus.cc:374] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:57:10.344139 3260 raft_consensus.cc:729] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f07077db5dbd405d968bf7b0da05e2ff, State: Initialized, Role: FOLLOWER
I20260925 22:57:10.344272 3260 consensus_queue.cc:260] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:10.344738 3260 ts_tablet_manager.cc:1428] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:57:10.346705 3181 tablet_service.cc:1467] Processing CreateTablet for tablet 36511de17eed42118a1aec6d5edc25a5 (DEFAULT_TABLE table=test-table [id=2d98669447e549a89c98989962b8ff31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:57:10.348701 3181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36511de17eed42118a1aec6d5edc25a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:57:10.348321 3049 tablet_service.cc:1467] Processing CreateTablet for tablet 36511de17eed42118a1aec6d5edc25a5 (DEFAULT_TABLE table=test-table [id=2d98669447e549a89c98989962b8ff31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:57:10.350490 3049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36511de17eed42118a1aec6d5edc25a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:57:10.358711 3263 tablet_bootstrap.cc:492] T 36511de17eed42118a1aec6d5edc25a5 P 28ef18fda7e441508e538985b7fdc630: Bootstrap starting.
I20260925 22:57:10.359395 3264 tablet_bootstrap.cc:492] T 36511de17eed42118a1aec6d5edc25a5 P 956e81b9add74bad987c4aa1d19ab4e9: Bootstrap starting.
I20260925 22:57:10.362309 3264 tablet_bootstrap.cc:654] T 36511de17eed42118a1aec6d5edc25a5 P 956e81b9add74bad987c4aa1d19ab4e9: Neither blocks nor log segments found. Creating new log.
I20260925 22:57:10.362309 3263 tablet_bootstrap.cc:654] T 36511de17eed42118a1aec6d5edc25a5 P 28ef18fda7e441508e538985b7fdc630: Neither blocks nor log segments found. Creating new log.
I20260925 22:57:10.363399 3263 log.cc:824] T 36511de17eed42118a1aec6d5edc25a5 P 28ef18fda7e441508e538985b7fdc630: Log is configured to *not* fsync() on all Append() calls
I20260925 22:57:10.363394 3264 log.cc:824] T 36511de17eed42118a1aec6d5edc25a5 P 956e81b9add74bad987c4aa1d19ab4e9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:57:10.366118 3264 tablet_bootstrap.cc:492] T 36511de17eed42118a1aec6d5edc25a5 P 956e81b9add74bad987c4aa1d19ab4e9: No bootstrap required, opened a new log
I20260925 22:57:10.366118 3263 tablet_bootstrap.cc:492] T 36511de17eed42118a1aec6d5edc25a5 P 28ef18fda7e441508e538985b7fdc630: No bootstrap required, opened a new log
I20260925 22:57:10.366384 3263 ts_tablet_manager.cc:1397] T 36511de17eed42118a1aec6d5edc25a5 P 28ef18fda7e441508e538985b7fdc630: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 22:57:10.366384 3264 ts_tablet_manager.cc:1397] T 36511de17eed42118a1aec6d5edc25a5 P 956e81b9add74bad987c4aa1d19ab4e9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:57:10.371851 3263 raft_consensus.cc:348] T 36511de17eed42118a1aec6d5edc25a5 P 28ef18fda7e441508e538985b7fdc630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:10.371851 3264 raft_consensus.cc:348] T 36511de17eed42118a1aec6d5edc25a5 P 956e81b9add74bad987c4aa1d19ab4e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:10.372360 3264 raft_consensus.cc:374] T 36511de17eed42118a1aec6d5edc25a5 P 956e81b9add74bad987c4aa1d19ab4e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:57:10.372360 3263 raft_consensus.cc:374] T 36511de17eed42118a1aec6d5edc25a5 P 28ef18fda7e441508e538985b7fdc630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:57:10.372460 3263 raft_consensus.cc:729] T 36511de17eed42118a1aec6d5edc25a5 P 28ef18fda7e441508e538985b7fdc630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28ef18fda7e441508e538985b7fdc630, State: Initialized, Role: FOLLOWER
I20260925 22:57:10.372460 3264 raft_consensus.cc:729] T 36511de17eed42118a1aec6d5edc25a5 P 956e81b9add74bad987c4aa1d19ab4e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 956e81b9add74bad987c4aa1d19ab4e9, State: Initialized, Role: FOLLOWER
I20260925 22:57:10.373579 3264 consensus_queue.cc:260] T 36511de17eed42118a1aec6d5edc25a5 P 956e81b9add74bad987c4aa1d19ab4e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:10.373579 3263 consensus_queue.cc:260] T 36511de17eed42118a1aec6d5edc25a5 P 28ef18fda7e441508e538985b7fdc630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:10.374508 3247 heartbeater.cc:503] Master 127.20.114.254:43183 was elected leader, sending a full tablet report...
I20260925 22:57:10.375087 3264 ts_tablet_manager.cc:1428] T 36511de17eed42118a1aec6d5edc25a5 P 956e81b9add74bad987c4aa1d19ab4e9: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260925 22:57:10.375092 3263 ts_tablet_manager.cc:1428] T 36511de17eed42118a1aec6d5edc25a5 P 28ef18fda7e441508e538985b7fdc630: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260925 22:57:10.375521 3115 heartbeater.cc:503] Master 127.20.114.254:43183 was elected leader, sending a full tablet report...
I20260925 22:57:10.443773 3262 raft_consensus.cc:482] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:57:10.444097 3262 raft_consensus.cc:504] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:10.445693 3262 leader_election.cc:288] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28ef18fda7e441508e538985b7fdc630 (127.20.114.195:43563), 956e81b9add74bad987c4aa1d19ab4e9 (127.20.114.194:45971)
I20260925 22:57:10.451347 3258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:10.446494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:33771 (local address 127.20.114.194:45971)
0925 22:57:10.446682 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:10.446686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:10.446702 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:57:10.446762 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:10.446764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:10.446821 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:10.446972 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:57:10.447311 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.448168 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.450027 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.450572 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.450766 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:10.450830 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:10.451132 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 22:57:10.451217 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 22:57:10.451339 3259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:10.446086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:35821 (local address 127.20.114.195:43563)
0925 22:57:10.446239 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:10.446244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:10.446388 (+ 144us) server_negotiation.cc:407] Connection header received
0925 22:57:10.446634 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:10.446638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:10.446716 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:10.446804 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:57:10.447264 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.448128 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.450041 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.450582 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.450792 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:10.450849 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:10.451104 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 22:57:10.451188 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:57:10.451361 3269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:10.445967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:43563 (local address 127.20.114.193:35821)
0925 22:57:10.446261 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:10.446279 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:10.446435 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:10.446842 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:10.446847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:10.446905 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:57:10.447094 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:10.447102 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.448258 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.448265 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:10.449818 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:10.449832 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.450786 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.450790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:10.450947 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:10.450985 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 22:57:10.451053 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 22:57:10.451177 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":466,"thread_start_us":124,"threads_started":1}
I20260925 22:57:10.451378 3270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:10.446024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:45971 (local address 127.20.114.193:33771)
0925 22:57:10.446360 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:10.446371 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:10.446443 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:10.447088 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:10.447091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:10.447110 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:57:10.447207 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:10.447215 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.448290 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.448293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:10.449812 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:10.449829 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.450783 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.450787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:10.450933 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:10.450980 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 22:57:10.451094 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 22:57:10.451180 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":100,"thread_start_us":119,"threads_started":1}
I20260925 22:57:10.452099 3201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36511de17eed42118a1aec6d5edc25a5" candidate_uuid: "f07077db5dbd405d968bf7b0da05e2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28ef18fda7e441508e538985b7fdc630" is_pre_election: true
I20260925 22:57:10.452104 3069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36511de17eed42118a1aec6d5edc25a5" candidate_uuid: "f07077db5dbd405d968bf7b0da05e2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "956e81b9add74bad987c4aa1d19ab4e9" is_pre_election: true
I20260925 22:57:10.452443 3069 raft_consensus.cc:2454] T 36511de17eed42118a1aec6d5edc25a5 P 956e81b9add74bad987c4aa1d19ab4e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f07077db5dbd405d968bf7b0da05e2ff in term 0.
I20260925 22:57:10.452445 3201 raft_consensus.cc:2454] T 36511de17eed42118a1aec6d5edc25a5 P 28ef18fda7e441508e538985b7fdc630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f07077db5dbd405d968bf7b0da05e2ff in term 0.
I20260925 22:57:10.452978 2865 leader_election.cc:302] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28ef18fda7e441508e538985b7fdc630, f07077db5dbd405d968bf7b0da05e2ff; no voters:
I20260925 22:57:10.453295 3262 raft_consensus.cc:2781] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:57:10.453392 3262 raft_consensus.cc:482] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:57:10.453434 3262 raft_consensus.cc:3037] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:57:10.454486 3262 raft_consensus.cc:504] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:10.454901 3262 leader_election.cc:288] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 1 election: Requested vote from peers 28ef18fda7e441508e538985b7fdc630 (127.20.114.195:43563), 956e81b9add74bad987c4aa1d19ab4e9 (127.20.114.194:45971)
I20260925 22:57:10.455358 3201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36511de17eed42118a1aec6d5edc25a5" candidate_uuid: "f07077db5dbd405d968bf7b0da05e2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28ef18fda7e441508e538985b7fdc630"
I20260925 22:57:10.455518 3201 raft_consensus.cc:3037] T 36511de17eed42118a1aec6d5edc25a5 P 28ef18fda7e441508e538985b7fdc630 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:57:10.455575 3069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36511de17eed42118a1aec6d5edc25a5" candidate_uuid: "f07077db5dbd405d968bf7b0da05e2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "956e81b9add74bad987c4aa1d19ab4e9"
I20260925 22:57:10.455849 3069 raft_consensus.cc:3037] T 36511de17eed42118a1aec6d5edc25a5 P 956e81b9add74bad987c4aa1d19ab4e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:57:10.457530 3069 raft_consensus.cc:2454] T 36511de17eed42118a1aec6d5edc25a5 P 956e81b9add74bad987c4aa1d19ab4e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f07077db5dbd405d968bf7b0da05e2ff in term 1.
I20260925 22:57:10.457567 3201 raft_consensus.cc:2454] T 36511de17eed42118a1aec6d5edc25a5 P 28ef18fda7e441508e538985b7fdc630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f07077db5dbd405d968bf7b0da05e2ff in term 1.
I20260925 22:57:10.457962 2865 leader_election.cc:302] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28ef18fda7e441508e538985b7fdc630, f07077db5dbd405d968bf7b0da05e2ff; no voters:
I20260925 22:57:10.458231 3262 raft_consensus.cc:2781] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:57:10.458637 3262 raft_consensus.cc:686] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 1 LEADER]: Becoming Leader. State: Replica: f07077db5dbd405d968bf7b0da05e2ff, State: Running, Role: LEADER
I20260925 22:57:10.458940 3262 consensus_queue.cc:237] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:10.462675 2786 catalog_manager.cc:5495] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff reported cstate change: term changed from 0 to 1, leader changed from <none> to f07077db5dbd405d968bf7b0da05e2ff (127.20.114.193). New cstate: current_term: 1 leader_uuid: "f07077db5dbd405d968bf7b0da05e2ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } health_report { overall_health: HEALTHY } } }
I20260925 22:57:10.506178 3273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:10.501562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45498 (local address 127.20.114.193:39053)
0925 22:57:10.501981 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:10.501988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:10.502002 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:57:10.502123 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:10.502125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:10.502184 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:10.502295 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:57:10.502580 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.503157 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.504232 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:10.504487 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:10.504583 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:10.505059 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:57:10.505606 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:57:10.505706 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:10.505852 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 22:57:10.505914 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":139,"threads_started":1}
I20260925 22:57:10.511193 3201 raft_consensus.cc:1261] T 36511de17eed42118a1aec6d5edc25a5 P 28ef18fda7e441508e538985b7fdc630 [term 1 FOLLOWER]: Refusing update from remote peer f07077db5dbd405d968bf7b0da05e2ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:57:10.511214 3069 raft_consensus.cc:1261] T 36511de17eed42118a1aec6d5edc25a5 P 956e81b9add74bad987c4aa1d19ab4e9 [term 1 FOLLOWER]: Refusing update from remote peer f07077db5dbd405d968bf7b0da05e2ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:57:10.512084 3271 consensus_queue.cc:1035] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:57:10.512418 3262 consensus_queue.cc:1035] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:57:10.520231 3274 mvcc.cc:205] Tried to move back new op lower bound from 7333384316966232064 to 7333384316763521024. Current Snapshot: MvccSnapshot[applied={T|T < 7333384316966232064}]
I20260925 22:57:10.522279 3275 mvcc.cc:205] Tried to move back new op lower bound from 7333384316966232064 to 7333384316763521024. Current Snapshot: MvccSnapshot[applied={T|T < 7333384316966232064}]
I20260925 22:57:10.525501 3276 mvcc.cc:205] Tried to move back new op lower bound from 7333384316966232064 to 7333384316763521024. Current Snapshot: MvccSnapshot[applied={T|T < 7333384316966232064}]
I20260925 22:57:10.533205 2786 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:52948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:57:10.533423 2786 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:52948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:57:10.536401 2786 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2: Sending DeleteTablet for 3 replicas of tablet 36511de17eed42118a1aec6d5edc25a5
I20260925 22:57:10.537444 2911 tablet_service.cc:1514] Processing DeleteTablet for tablet 36511de17eed42118a1aec6d5edc25a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:57:10 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:45488
I20260925 22:57:10.537395 3049 tablet_service.cc:1514] Processing DeleteTablet for tablet 36511de17eed42118a1aec6d5edc25a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:57:10 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:51392
I20260925 22:57:10.537781 3181 tablet_service.cc:1514] Processing DeleteTablet for tablet 36511de17eed42118a1aec6d5edc25a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:57:10 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:39458
I20260925 22:57:10.538309 3285 tablet_replica.cc:330] T 36511de17eed42118a1aec6d5edc25a5 P 28ef18fda7e441508e538985b7fdc630: stopping tablet replica
I20260925 22:57:10.538655 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 2827
I20260925 22:57:10.538884 3285 raft_consensus.cc:2229] T 36511de17eed42118a1aec6d5edc25a5 P 28ef18fda7e441508e538985b7fdc630 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:57:10.539446 3285 raft_consensus.cc:2258] T 36511de17eed42118a1aec6d5edc25a5 P 28ef18fda7e441508e538985b7fdc630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:57:10.539469 3284 tablet_replica.cc:330] T 36511de17eed42118a1aec6d5edc25a5 P 956e81b9add74bad987c4aa1d19ab4e9: stopping tablet replica
I20260925 22:57:10.539812 3284 raft_consensus.cc:2229] T 36511de17eed42118a1aec6d5edc25a5 P 956e81b9add74bad987c4aa1d19ab4e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:57:10.540495 3284 raft_consensus.cc:2258] T 36511de17eed42118a1aec6d5edc25a5 P 956e81b9add74bad987c4aa1d19ab4e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:57:10.541373 3285 ts_tablet_manager.cc:1905] T 36511de17eed42118a1aec6d5edc25a5 P 28ef18fda7e441508e538985b7fdc630: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:57:10.542654 3284 ts_tablet_manager.cc:1905] T 36511de17eed42118a1aec6d5edc25a5 P 956e81b9add74bad987c4aa1d19ab4e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:57:10.544086 3285 ts_tablet_manager.cc:1918] T 36511de17eed42118a1aec6d5edc25a5 P 28ef18fda7e441508e538985b7fdc630: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:57:10.544222 3285 log.cc:1196] T 36511de17eed42118a1aec6d5edc25a5 P 28ef18fda7e441508e538985b7fdc630: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/36511de17eed42118a1aec6d5edc25a5
I20260925 22:57:10.544535 3285 ts_tablet_manager.cc:1939] T 36511de17eed42118a1aec6d5edc25a5 P 28ef18fda7e441508e538985b7fdc630: Deleting consensus metadata
I20260925 22:57:10.546731 3284 ts_tablet_manager.cc:1918] T 36511de17eed42118a1aec6d5edc25a5 P 956e81b9add74bad987c4aa1d19ab4e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:57:10.546948 3284 log.cc:1196] T 36511de17eed42118a1aec6d5edc25a5 P 956e81b9add74bad987c4aa1d19ab4e9: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/36511de17eed42118a1aec6d5edc25a5
I20260925 22:57:10.547098 2775 catalog_manager.cc:4841] TS 28ef18fda7e441508e538985b7fdc630 (127.20.114.195:43563): tablet 36511de17eed42118a1aec6d5edc25a5 (table test-table [id=2d98669447e549a89c98989962b8ff31]) successfully deleted
I20260925 22:57:10.547385 3284 ts_tablet_manager.cc:1939] T 36511de17eed42118a1aec6d5edc25a5 P 956e81b9add74bad987c4aa1d19ab4e9: Deleting consensus metadata
W20260925 22:57:10.547741 2773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.114.193:39053 (error 108)
I20260925 22:57:10.548386 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 2982
I20260925 22:57:10.549036 3257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:10.548614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:39053 (local address 127.0.0.1:45514)
0925 22:57:10.548827 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:10.548932 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.193:39053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260925 22:57:10.549268 2773 catalog_manager.cc:4584] TS f07077db5dbd405d968bf7b0da05e2ff (127.20.114.193:39053): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 36511de17eed42118a1aec6d5edc25a5: Network error: Client connection negotiation failed: client connection to 127.20.114.193:39053: connect: Connection refused (error 111)
I20260925 22:57:10.556046 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 3118
I20260925 22:57:10.558147 3257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:10.557822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:45971 (local address 127.0.0.1:51408)
0925 22:57:10.557978 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:10.558050 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.194:45971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 22:57:10.562148 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 2756
I20260925 22:57:10.570791 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:43183
--webserver_interface=127.20.114.254
--webserver_port=44623
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:43183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:57:10.712391 3287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:10.712730 3287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:10.712774 3287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:10.718154 3287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:57:10.718256 3287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:10.718278 3287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:10.718297 3287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:57:10.718314 3287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:57:10.725059 3287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:43183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:43183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=44623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:10.726606 3287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:10.727851 3287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:57:10.738981 3292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:10.738883 3293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:10.738971 3287 server_base.cc:1034] running on GCE node
W20260925 22:57:10.739056 3295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:10.739959 3287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:10.741580 3287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:10.742786 3287 hybrid_clock.cc:648] HybridClock initialized: now 1790377030742752 us; error 51 us; skew 500 ppm
Sep 25 22:57:10 dist-test-slave-x1lh krb5kdc[2208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377030, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:10.746265 3287 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:57:10.747598 3287 webserver.cc:460] Webserver started at http://127.20.114.254:44623/ using document root <none> and password file <none>
I20260925 22:57:10.748374 3287 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:10.748720 3287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:10.754771 3287 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 22:57:10.758366 3302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:10.760068 3287 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:57:10.760290 3287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "d3c132d8718d473c94d4fe821ec671f2"
format_stamp: "Formatted at 2026-09-25 22:57:07 on dist-test-slave-x1lh"
server_key: "66b48594ac034fb968b6c47da26d1ed8"
server_key_iv: "1609fa952a5439cb8ceb8caf2f9a099e"
server_key_version: "encryptionkey@0"
I20260925 22:57:10.761022 3287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:10.784660 3287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:10.789948 3287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:10.790256 3287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:10.798866 3287 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:43183
I20260925 22:57:10.799180 3354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:43183 every 8 connection(s)
I20260925 22:57:10.800173 3287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:57:10.801072 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 3287
I20260925 22:57:10.801748 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:39053
--local_ip_for_outbound_sockets=127.20.114.193
--tserver_master_addrs=127.20.114.254:43183
--webserver_port=41425
--webserver_interface=127.20.114.193
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:57:10.806895 3355 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:57:10.809397 3355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2: Bootstrap starting.
I20260925 22:57:10.830590 3355 log.cc:824] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2: Log is configured to *not* fsync() on all Append() calls
I20260925 22:57:10.843833 3355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:57:10.844506 3355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2: Bootstrap complete.
I20260925 22:57:10.849543 3355 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c132d8718d473c94d4fe821ec671f2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43183 } }
I20260925 22:57:10.850741 3355 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3c132d8718d473c94d4fe821ec671f2, State: Initialized, Role: FOLLOWER
I20260925 22:57:10.851610 3355 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c132d8718d473c94d4fe821ec671f2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43183 } }
I20260925 22:57:10.851806 3355 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:57:10.851902 3355 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:57:10.852034 3355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 2 FOLLOWER]: Advancing to term 3
I20260925 22:57:10.854382 3355 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c132d8718d473c94d4fe821ec671f2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43183 } }
I20260925 22:57:10.854933 3355 leader_election.cc:302] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3c132d8718d473c94d4fe821ec671f2; no voters:
I20260925 22:57:10.855358 3355 leader_election.cc:288] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 22:57:10.855463 3359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 22:57:10.855675 3359 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [term 3 LEADER]: Becoming Leader. State: Replica: d3c132d8718d473c94d4fe821ec671f2, State: Running, Role: LEADER
I20260925 22:57:10.856012 3359 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c132d8718d473c94d4fe821ec671f2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43183 } }
I20260925 22:57:10.856923 3355 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:57:10.857755 3360 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d3c132d8718d473c94d4fe821ec671f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c132d8718d473c94d4fe821ec671f2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43183 } } }
I20260925 22:57:10.857803 3361 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3c132d8718d473c94d4fe821ec671f2. Latest consensus state: current_term: 3 leader_uuid: "d3c132d8718d473c94d4fe821ec671f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c132d8718d473c94d4fe821ec671f2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 43183 } } }
I20260925 22:57:10.857901 3360 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:57:10.857910 3361 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c132d8718d473c94d4fe821ec671f2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:57:10.858327 3363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:57:10.866494 3363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:57:10.867796 3363 catalog_manager.cc:1259] Loaded cluster ID: 3512b03610ba46fab4de6dc9b45d6f14
I20260925 22:57:10.867915 3363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 22:57:10.869763 3363 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4e9782f9ad google::LogMessage::Fail() at ??:0
@ 0x7f4e9783288a google::LogMessage::SendToLog() at ??:0
@ 0x7f4e9782f4de google::LogMessage::Flush() at ??:0
@ 0x7f4e97830a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4ea216efdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f4ea216fd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4ea216c579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f4ea21b0640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4e99a813d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4e99a81c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4e99a833f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4e99a72512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4e9b0b66db start_thread at ??:0
@ 0x7f4e9632d61f clone at ??:0
W20260925 22:57:10.974675 3357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:10.975147 3357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:10.975229 3357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:10.983763 3357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:10.983887 3357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:10.983982 3357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:57:10.995095 3357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:39053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=41425
--tserver_master_addrs=127.20.114.254:43183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:10.996865 3357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:10.998608 3357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:57:11.014739 3396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:11.014919 3357 server_base.cc:1034] running on GCE node
W20260925 22:57:11.014693 3397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:11.014739 3399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:11.015779 3357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:11.016741 3357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:11.017971 3357 hybrid_clock.cc:648] HybridClock initialized: now 1790377031017948 us; error 33 us; skew 500 ppm
Sep 25 22:57:11 dist-test-slave-x1lh krb5kdc[2208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377031, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:11.022745 3357 init.cc:377] Logged in from keytab as kudu/127.20.114.193@KRBTEST.COM (short username kudu)
I20260925 22:57:11.024331 3357 webserver.cc:460] Webserver started at http://127.20.114.193:41425/ using document root <none> and password file <none>
I20260925 22:57:11.025281 3357 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:11.025481 3357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:11.033380 3357 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 22:57:11.037101 3406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:11.039039 3357 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:57:11.039222 3357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "f07077db5dbd405d968bf7b0da05e2ff"
format_stamp: "Formatted at 2026-09-25 22:57:08 on dist-test-slave-x1lh"
server_key: "091208c7f52b0b69e7a6964c3e9cb66d"
server_key_iv: "3cf1ccd24414325ba917f4a7ea395ea1"
server_key_version: "encryptionkey@0"
I20260925 22:57:11.040098 3357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:11.079633 3357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:11.085366 3357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:11.085668 3357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:11.086654 3357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:57:11.088632 3413 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:57:11.093657 3357 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:57:11.093771 3357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260925 22:57:11.093849 3357 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:57:11.096386 3357 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:57:11.096469 3357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260925 22:57:11.097023 3413 tablet_bootstrap.cc:492] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff: Bootstrap starting.
I20260925 22:57:11.110241 3357 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:39053
I20260925 22:57:11.110675 3520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:39053 every 8 connection(s)
I20260925 22:57:11.111869 3357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:57:11.112528 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 3357
I20260925 22:57:11.113145 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:45971
--local_ip_for_outbound_sockets=127.20.114.194
--tserver_master_addrs=127.20.114.254:43183
--webserver_port=39915
--webserver_interface=127.20.114.194
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:57:11.122480 3413 log.cc:824] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff: Log is configured to *not* fsync() on all Append() calls
I20260925 22:57:11.132200 3413 tablet_bootstrap.cc:492] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:57:11.132674 3413 tablet_bootstrap.cc:492] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff: Bootstrap complete.
I20260925 22:57:11.134338 3413 ts_tablet_manager.cc:1397] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff: Time spent bootstrapping tablet: real 0.038s user 0.029s sys 0.007s
I20260925 22:57:11.137514 3413 raft_consensus.cc:348] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:11.138032 3413 raft_consensus.cc:729] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f07077db5dbd405d968bf7b0da05e2ff, State: Initialized, Role: FOLLOWER
I20260925 22:57:11.138819 3413 consensus_queue.cc:260] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:11.139937 3413 ts_tablet_manager.cc:1428] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260925 22:57:11.235646 3524 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:57:11.116475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.193:33145)
0925 22:57:11.117334 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:11.117404 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:11.119039 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:11.234568 (+115529us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":227,"threads_started":1}
W20260925 22:57:11.236496 3521 heartbeater.cc:644] Failed to heartbeat to 127.20.114.254:43183 (0 consecutive failures): Network error: Failed to ping master at 127.20.114.254:43183: Client connection negotiation failed: client connection to 127.20.114.254:43183: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 22:57:11.238349 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:11.237782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.193:56663)
0925 22:57:11.238188 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:11.238270 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286}
I20260925 22:57:11.239239 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:11.238965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.193:39349)
0925 22:57:11.239125 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:11.239176 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260925 22:57:11.239488 3521 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:57:11.291703 3523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:11.292161 3523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:11.292289 3523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:11.297072 3523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:11.297173 3523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:11.297251 3523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:57:11.304951 3523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:45971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=39915
--tserver_master_addrs=127.20.114.254:43183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:11.306631 3523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:11.307853 3523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:57:11.321141 3531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:11.321197 3532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:11.321434 3523 server_base.cc:1034] running on GCE node
W20260925 22:57:11.321740 3534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:11.322266 3523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:11.323055 3523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:11.324282 3523 hybrid_clock.cc:648] HybridClock initialized: now 1790377031324248 us; error 64 us; skew 500 ppm
Sep 25 22:57:11 dist-test-slave-x1lh krb5kdc[2208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377031, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:11.327543 3523 init.cc:377] Logged in from keytab as kudu/127.20.114.194@KRBTEST.COM (short username kudu)
I20260925 22:57:11.328531 3523 webserver.cc:460] Webserver started at http://127.20.114.194:39915/ using document root <none> and password file <none>
I20260925 22:57:11.329201 3523 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:11.329275 3523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:11.334779 3523 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260925 22:57:11.338120 3541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:11.340452 3523 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:57:11.340682 3523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "956e81b9add74bad987c4aa1d19ab4e9"
format_stamp: "Formatted at 2026-09-25 22:57:08 on dist-test-slave-x1lh"
server_key: "0fe5227eaa09142190666b24a221c256"
server_key_iv: "16dfdc12d09536d911f45728e4bfd5f2"
server_key_version: "encryptionkey@0"
I20260925 22:57:11.341460 3523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:11.380209 3523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:11.384326 3523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:11.384563 3523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:11.385324 3523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:57:11.386266 3523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:57:11.386336 3523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:11.386404 3523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:57:11.386444 3523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:11.399976 3523 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:45971
I20260925 22:57:11.400257 3654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:45971 every 8 connection(s)
I20260925 22:57:11.401283 3523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:57:11.406688 3657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:11.404283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.194:55469)
0925 22:57:11.405085 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:11.405618 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":136,"threads_started":1}
I20260925 22:57:11.406843 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 3523
W20260925 22:57:11.407294 3655 heartbeater.cc:644] Failed to heartbeat to 127.20.114.254:43183 (0 consecutive failures): Network error: Failed to ping master at 127.20.114.254:43183: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
I20260925 22:57:11.407500 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:43563
--local_ip_for_outbound_sockets=127.20.114.195
--tserver_master_addrs=127.20.114.254:43183
--webserver_port=34645
--webserver_interface=127.20.114.195
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:57:11.408174 3657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:11.407810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.194:55965)
0925 22:57:11.407980 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:11.408066 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 22:57:11.409117 3657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:11.408754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.194:58673)
0925 22:57:11.408967 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:11.409038 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260925 22:57:11.409364 3655 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:57:11.555593 3658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:11.555934 3658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:11.556016 3658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:11.560421 3658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:11.560542 3658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:11.560642 3658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:57:11.567126 3658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:43563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=34645
--tserver_master_addrs=127.20.114.254:43183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:11.568401 3658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:11.569581 3658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:57:11.582409 3663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:11.582623 3666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:11.582417 3664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:11.583127 3658 server_base.cc:1034] running on GCE node
I20260925 22:57:11.583957 3658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:11.584839 3658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:11.586227 3658 hybrid_clock.cc:648] HybridClock initialized: now 1790377031586188 us; error 117 us; skew 500 ppm
Sep 25 22:57:11 dist-test-slave-x1lh krb5kdc[2208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377031, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:11.591279 3658 init.cc:377] Logged in from keytab as kudu/127.20.114.195@KRBTEST.COM (short username kudu)
I20260925 22:57:11.592533 3658 webserver.cc:460] Webserver started at http://127.20.114.195:34645/ using document root <none> and password file <none>
I20260925 22:57:11.593360 3658 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:11.593479 3658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:11.598769 3658 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:57:11.601686 3673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:11.603390 3658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:57:11.603593 3658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "28ef18fda7e441508e538985b7fdc630"
format_stamp: "Formatted at 2026-09-25 22:57:08 on dist-test-slave-x1lh"
server_key: "2f59199e402207f35ca0849c8718ebdc"
server_key_iv: "9805d192c6518ec0a25a047058eb83b6"
server_key_version: "encryptionkey@0"
I20260925 22:57:11.604306 3658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:11.626508 3658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:11.631011 3658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:11.631270 3658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:11.631982 3658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:57:11.633064 3658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:57:11.633136 3658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:11.633210 3658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:57:11.633249 3658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:11.645092 3658 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:43563
I20260925 22:57:11.645123 3786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:43563 every 8 connection(s)
I20260925 22:57:11.646298 3658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:57:11.646790 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 3658
I20260925 22:57:11.651974 3790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:11.649418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.195:44535)
0925 22:57:11.650470 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:11.650657 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":151,"threads_started":1}
W20260925 22:57:11.652590 3787 heartbeater.cc:644] Failed to heartbeat to 127.20.114.254:43183 (0 consecutive failures): Network error: Failed to ping master at 127.20.114.254:43183: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
I20260925 22:57:11.653436 3790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:11.653177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.195:53957)
0925 22:57:11.653307 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:11.653371 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 22:57:11.654313 3790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:11.654063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.195:34593)
0925 22:57:11.654189 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:11.654255 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 22:57:11.654629 3787 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 22:57:12.240583 3791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:12.240122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.193:53295)
0925 22:57:12.240416 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:12.240480 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":90,"threads_started":1}
I20260925 22:57:12.410463 3792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:12.409928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.194:57373)
0925 22:57:12.410233 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:12.410308 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":71,"threads_started":1}
I20260925 22:57:12.595773 3793 raft_consensus.cc:482] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:57:12.596130 3793 raft_consensus.cc:504] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:12.597558 3793 leader_election.cc:288] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28ef18fda7e441508e538985b7fdc630 (127.20.114.195:43563), 956e81b9add74bad987c4aa1d19ab4e9 (127.20.114.194:45971)
Sep 25 22:57:12 dist-test-slave-x1lh krb5kdc[2208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377031, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.195@KRBTEST.COM
Sep 25 22:57:12 dist-test-slave-x1lh krb5kdc[2208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377031, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.194@KRBTEST.COM
I20260925 22:57:12.614348 3796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:12.598388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:44017 (local address 127.20.114.194:45971)
0925 22:57:12.598745 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:12.598764 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:12.598800 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:57:12.599516 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:12.599523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:12.599905 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:12.600247 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:12.602242 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:12.603376 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:12.604673 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:12.605272 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:12.605470 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:12.608921 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:12.608931 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:12.608950 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:12.608992 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:12.610942 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:12.611735 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:12.611750 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:12.611755 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:12.611904 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:12.612538 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:12.612541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:12.612542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:12.613190 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:12.613333 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:12.613812 (+ 479us) server_negotiation.cc:299] Negotiation successful
0925 22:57:12.614131 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":127,"threads_started":1}
I20260925 22:57:12.614355 3795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:12.597982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:35807 (local address 127.20.114.195:43563)
0925 22:57:12.598697 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:12.598710 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:12.598750 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:57:12.599466 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:12.599473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:12.599905 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:12.600255 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:12.602376 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:12.603448 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:12.604753 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:12.605313 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:12.605510 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:12.608528 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:12.608534 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:12.608549 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:12.608592 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:12.610920 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:12.611671 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:12.611678 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:12.611683 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:12.611827 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:12.612321 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:12.612328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:12.612332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:12.613011 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:12.613143 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:12.613851 (+ 708us) server_negotiation.cc:299] Negotiation successful
0925 22:57:12.614146 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":95,"threads_started":1}
I20260925 22:57:12.614344 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:12.597904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:45971 (local address 127.20.114.193:44017)
0925 22:57:12.598466 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:12.598482 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:12.598588 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:12.600692 (+ 2104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:12.600695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:12.601297 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:12.601986 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:12.602008 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:12.603567 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:12.603576 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:12.604519 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:12.604530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:12.605417 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:12.605421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:12.605666 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:12.606414 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:12.606441 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:12.608581 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:12.611073 (+ 2492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:12.611086 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:12.611095 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:12.611521 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:12.612157 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:12.612161 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:12.612163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:12.612302 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:12.613311 (+ 1009us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:12.613313 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:12.613501 (+ 188us) client_negotiation.cc:769] Sending connection context
0925 22:57:12.613674 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 22:57:12.614096 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":255,"spinlock_wait_cycles":86272,"thread_start_us":80,"threads_started":1}
I20260925 22:57:12.614359 3791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:12.597780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:43563 (local address 127.20.114.193:35807)
0925 22:57:12.597977 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:12.597994 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:12.598112 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:12.600398 (+ 2286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:12.600405 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:12.601236 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:12.601986 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:12.602008 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:12.603750 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:12.603755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:12.604599 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:12.604610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:12.605447 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:12.605451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:12.605663 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:12.606427 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:12.606446 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:12.608224 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:12.611071 (+ 2847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:12.611079 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:12.611095 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:12.611520 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:12.611936 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:12.611939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:12.611941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:12.612196 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:12.613148 (+ 952us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:12.613156 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:12.613479 (+ 323us) client_negotiation.cc:769] Sending connection context
0925 22:57:12.613685 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 22:57:12.614182 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":307,"spinlock_wait_cycles":20224}
W20260925 22:57:12.617288 3410 leader_election.cc:341] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 956e81b9add74bad987c4aa1d19ab4e9 (127.20.114.194:45971): Not found: Tablet not found: 36511de17eed42118a1aec6d5edc25a5
W20260925 22:57:12.618139 3409 leader_election.cc:341] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28ef18fda7e441508e538985b7fdc630 (127.20.114.195:43563): Not found: Tablet not found: 36511de17eed42118a1aec6d5edc25a5
I20260925 22:57:12.618263 3409 leader_election.cc:302] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f07077db5dbd405d968bf7b0da05e2ff; no voters: 28ef18fda7e441508e538985b7fdc630, 956e81b9add74bad987c4aa1d19ab4e9
I20260925 22:57:12.618687 3793 raft_consensus.cc:2726] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:57:12.655666 3797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:12.655223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.195:44337)
0925 22:57:12.655499 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:12.655561 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":70,"threads_started":1}
I20260925 22:57:13.241894 3798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:13.241474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.193:48897)
0925 22:57:13.241733 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:13.241796 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":80,"threads_started":1}
I20260925 22:57:13.412005 3799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:13.411457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.194:39305)
0925 22:57:13.411829 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:13.411900 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":105,"threads_started":1}
I20260925 22:57:13.657214 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:13.656657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.195:58325)
0925 22:57:13.657027 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:13.657099 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":82,"threads_started":1}
I20260925 22:57:14.243359 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:14.242854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.193:36531)
0925 22:57:14.243196 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:14.243261 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":93,"threads_started":1}
I20260925 22:57:14.413769 3802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:14.413070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.194:42509)
0925 22:57:14.413420 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:14.413494 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":86,"threads_started":1}
I20260925 22:57:14.524274 3803 raft_consensus.cc:482] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:57:14.524426 3803 raft_consensus.cc:504] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:14.524955 3803 leader_election.cc:288] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28ef18fda7e441508e538985b7fdc630 (127.20.114.195:43563), 956e81b9add74bad987c4aa1d19ab4e9 (127.20.114.194:45971)
W20260925 22:57:14.526247 3410 leader_election.cc:341] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 956e81b9add74bad987c4aa1d19ab4e9 (127.20.114.194:45971): Not found: Tablet not found: 36511de17eed42118a1aec6d5edc25a5
W20260925 22:57:14.526427 3409 leader_election.cc:341] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28ef18fda7e441508e538985b7fdc630 (127.20.114.195:43563): Not found: Tablet not found: 36511de17eed42118a1aec6d5edc25a5
I20260925 22:57:14.526541 3409 leader_election.cc:302] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f07077db5dbd405d968bf7b0da05e2ff; no voters: 28ef18fda7e441508e538985b7fdc630, 956e81b9add74bad987c4aa1d19ab4e9
I20260925 22:57:14.526733 3803 raft_consensus.cc:2726] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:57:14.658965 3804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:14.658464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.195:49347)
0925 22:57:14.658777 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:14.658846 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":93,"threads_started":1}
I20260925 22:57:15.244735 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:15.244242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.193:42721)
0925 22:57:15.244582 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:15.244640 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":109,"threads_started":1}
I20260925 22:57:15.415887 3806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:15.415230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.194:52205)
0925 22:57:15.415658 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:15.415738 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":100,"threads_started":1}
I20260925 22:57:15.660522 3807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:15.660002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.195:34755)
0925 22:57:15.660348 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:15.660418 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":106,"threads_started":1}
I20260925 22:57:16.246469 3808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:16.245944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.193:35949)
0925 22:57:16.246306 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:16.246373 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":129,"threads_started":1}
I20260925 22:57:16.417536 3809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:16.417038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.194:46479)
0925 22:57:16.417340 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:16.417402 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":119,"threads_started":1}
I20260925 22:57:16.662032 3810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:16.661492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.195:48925)
0925 22:57:16.661810 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:16.661879 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":100,"threads_started":1}
I20260925 22:57:17.248027 3811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:17.247546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.193:55655)
0925 22:57:17.247838 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:17.247921 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":112,"threads_started":1}
I20260925 22:57:17.289247 3812 raft_consensus.cc:482] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:57:17.289402 3812 raft_consensus.cc:504] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:17.289844 3812 leader_election.cc:288] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28ef18fda7e441508e538985b7fdc630 (127.20.114.195:43563), 956e81b9add74bad987c4aa1d19ab4e9 (127.20.114.194:45971)
W20260925 22:57:17.290571 3409 leader_election.cc:341] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28ef18fda7e441508e538985b7fdc630 (127.20.114.195:43563): Not found: Tablet not found: 36511de17eed42118a1aec6d5edc25a5
W20260925 22:57:17.290704 3410 leader_election.cc:341] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 956e81b9add74bad987c4aa1d19ab4e9 (127.20.114.194:45971): Not found: Tablet not found: 36511de17eed42118a1aec6d5edc25a5
I20260925 22:57:17.290797 3410 leader_election.cc:302] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f07077db5dbd405d968bf7b0da05e2ff; no voters: 28ef18fda7e441508e538985b7fdc630, 956e81b9add74bad987c4aa1d19ab4e9
I20260925 22:57:17.290966 3812 raft_consensus.cc:2726] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:57:17.419019 3813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:17.418475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.194:57003)
0925 22:57:17.418776 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:17.418842 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":95,"threads_started":1}
I20260925 22:57:17.663585 3814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:17.663123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.195:38021)
0925 22:57:17.663412 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:17.663482 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":104,"threads_started":1}
I20260925 22:57:18.249536 3815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:18.249031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.193:39469)
0925 22:57:18.249327 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:18.249397 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":73,"threads_started":1}
I20260925 22:57:18.420313 3816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:18.419808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.194:35295)
0925 22:57:18.420136 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:18.420202 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":74,"threads_started":1}
I20260925 22:57:18.664991 3817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:18.664427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.195:53943)
0925 22:57:18.664771 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:18.664833 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":90,"threads_started":1}
I20260925 22:57:19.250896 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:19.250437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.193:48707)
0925 22:57:19.250733 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:19.250798 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":96,"threads_started":1}
I20260925 22:57:19.422056 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:19.421395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.194:33441)
0925 22:57:19.421811 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:19.421923 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":133,"threads_started":1}
I20260925 22:57:19.666321 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:19.665851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.195:39607)
0925 22:57:19.666155 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:19.666221 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":97,"threads_started":1}
I20260925 22:57:20.221196 3821 raft_consensus.cc:482] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:57:20.221341 3821 raft_consensus.cc:504] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ef18fda7e441508e538985b7fdc630" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 43563 } } peers { permanent_uuid: "956e81b9add74bad987c4aa1d19ab4e9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 45971 } } peers { permanent_uuid: "f07077db5dbd405d968bf7b0da05e2ff" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39053 } }
I20260925 22:57:20.221827 3821 leader_election.cc:288] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28ef18fda7e441508e538985b7fdc630 (127.20.114.195:43563), 956e81b9add74bad987c4aa1d19ab4e9 (127.20.114.194:45971)
W20260925 22:57:20.222627 3410 leader_election.cc:341] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 956e81b9add74bad987c4aa1d19ab4e9 (127.20.114.194:45971): Not found: Tablet not found: 36511de17eed42118a1aec6d5edc25a5
W20260925 22:57:20.222729 3409 leader_election.cc:341] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28ef18fda7e441508e538985b7fdc630 (127.20.114.195:43563): Not found: Tablet not found: 36511de17eed42118a1aec6d5edc25a5
I20260925 22:57:20.222849 3409 leader_election.cc:302] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f07077db5dbd405d968bf7b0da05e2ff; no voters: 28ef18fda7e441508e538985b7fdc630, 956e81b9add74bad987c4aa1d19ab4e9
I20260925 22:57:20.223037 3821 raft_consensus.cc:2726] T 36511de17eed42118a1aec6d5edc25a5 P f07077db5dbd405d968bf7b0da05e2ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:57:20.252316 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:20.251871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.193:39237)
0925 22:57:20.252158 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:20.252218 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":100,"threads_started":1}
I20260925 22:57:20.423415 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:20.422939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.194:33303)
0925 22:57:20.423239 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:20.423305 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":97,"threads_started":1}
I20260925 22:57:20.667806 3824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:20.667362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.195:48113)
0925 22:57:20.667642 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:20.667708 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":92,"threads_started":1}
I20260925 22:57:21.253973 3825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:21.253474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.193:45171)
0925 22:57:21.253806 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:21.253874 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":102,"threads_started":1}
I20260925 22:57:21.424978 3826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:21.424375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.194:45255)
0925 22:57:21.424722 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:21.424815 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":126,"threads_started":1}
I20260925 22:57:21.669332 3827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:21.668795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.195:56443)
0925 22:57:21.669148 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:21.669213 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":141,"threads_started":1}
I20260925 22:57:22.255443 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:22.254909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.193:52999)
0925 22:57:22.255275 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:22.255340 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":122,"threads_started":1}
I20260925 22:57:22.426247 3829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:22.425822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.194:58053)
0925 22:57:22.426088 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:22.426147 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":81,"threads_started":1}
I20260925 22:57:22.670957 3830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:22.670403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.195:44681)
0925 22:57:22.670753 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:22.670820 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":120,"threads_started":1}
I20260925 22:57:23.257056 3831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:23.256598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.193:38121)
0925 22:57:23.256883 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:23.256955 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":97,"threads_started":1}
I20260925 22:57:23.427553 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:23.427116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.194:43075)
0925 22:57:23.427388 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:23.427450 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
I20260925 22:57:23.672356 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:23.671832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.195:55045)
0925 22:57:23.672191 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:23.672255 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":149,"threads_started":1}
I20260925 22:57:24.258283 3834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:24.257839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.193:57659)
0925 22:57:24.258124 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:24.258187 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":80,"threads_started":1}
I20260925 22:57:24.429080 3835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:24.428511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.194:34061)
0925 22:57:24.428778 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:24.428843 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":81,"threads_started":1}
I20260925 22:57:24.673795 3836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:24.673321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.195:37043)
0925 22:57:24.673630 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:24.673695 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":79,"threads_started":1}
I20260925 22:57:25.259716 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:25.259203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.193:32899)
0925 22:57:25.259524 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:25.259613 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":77,"threads_started":1}
I20260925 22:57:25.430482 3838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:25.429996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.194:57985)
0925 22:57:25.430309 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:25.430382 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":91,"threads_started":1}
I20260925 22:57:25.675282 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:25.674781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.195:37689)
0925 22:57:25.675115 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:25.675180 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":94,"threads_started":1}
I20260925 22:57:26.261415 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:26.260849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.193:59265)
0925 22:57:26.261252 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:26.261321 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":76,"threads_started":1}
I20260925 22:57:26.432080 3841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:26.431600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:43183 (local address 127.20.114.194:42063)
0925 22:57:26.431869 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:26.431954 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
I20260925 22:57:26.657207 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 3357
I20260925 22:57:26.663756 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 3523
I20260925 22:57:26.670174 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 3658
2026-09-25T22:57:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (20274 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:57:26 dist-test-slave-x1lh krb5kdc[3844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:57:26 dist-test-slave-x1lh krb5kdc[3844](info): set up 2 sockets
Sep 25 22:57:26 dist-test-slave-x1lh krb5kdc[3844](info): commencing operation
krb5kdc: starting...
W20260925 22:57:27.753464 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:57:27 dist-test-slave-x1lh krb5kdc[3844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:57:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:57:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:57:27.906322 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:37499
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:37499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:57:28.033691 3857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:28.034080 3857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:28.034175 3857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:28.039543 3857 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 22:57:28.039702 3857 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 22:57:28.039734 3857 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 22:57:28.039757 3857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:57:28.039795 3857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:28.039839 3857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:28.039875 3857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:57:28.039914 3857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:57:28.046056 3857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:37499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:37499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:28.047359 3857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:28.048487 3857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:57:28.058324 3862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:28.058471 3857 server_base.cc:1034] running on GCE node
W20260925 22:57:28.058362 3863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:28.058362 3865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:28.059332 3857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:28.060521 3857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:28.061736 3857 hybrid_clock.cc:648] HybridClock initialized: now 1790377048061694 us; error 58 us; skew 500 ppm
Sep 25 22:57:28 dist-test-slave-x1lh krb5kdc[3844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377048, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:28.065150 3857 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:57:28.066269 3857 webserver.cc:460] Webserver started at http://127.20.114.254:40287/ using document root <none> and password file <none>
I20260925 22:57:28.066896 3857 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:28.066995 3857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:28.067257 3857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:57:28.069684 3857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "0351ec2f6f864dcfa85800eb53556115"
format_stamp: "Formatted at 2026-09-25 22:57:28 on dist-test-slave-x1lh"
server_key: "19120fcff5ef12ed91e7b80d854f1a31"
server_key_iv: "1efbb08296b3845f62b31883fcb9f014"
server_key_version: "encryptionkey@0"
I20260925 22:57:28.070386 3857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "0351ec2f6f864dcfa85800eb53556115"
format_stamp: "Formatted at 2026-09-25 22:57:28 on dist-test-slave-x1lh"
server_key: "19120fcff5ef12ed91e7b80d854f1a31"
server_key_iv: "1efbb08296b3845f62b31883fcb9f014"
server_key_version: "encryptionkey@0"
I20260925 22:57:28.074889 3857 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 22:57:28.078003 3872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:28.079598 3857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:57:28.079780 3857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "0351ec2f6f864dcfa85800eb53556115"
format_stamp: "Formatted at 2026-09-25 22:57:28 on dist-test-slave-x1lh"
server_key: "19120fcff5ef12ed91e7b80d854f1a31"
server_key_iv: "1efbb08296b3845f62b31883fcb9f014"
server_key_version: "encryptionkey@0"
I20260925 22:57:28.079922 3857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:28.093649 3857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:28.097400 3857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:28.097646 3857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:28.105530 3857 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:37499
I20260925 22:57:28.105823 3924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:37499 every 8 connection(s)
I20260925 22:57:28.106608 3857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:57:28.109773 3925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:57:28.114579 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 3857
I20260925 22:57:28.114720 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:57:28.115034 20939 external_mini_cluster.cc:1428] Setting key 333825e5dfc538c7bbcd9227af65301b
I20260925 22:57:28.116532 3925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115: Bootstrap starting.
I20260925 22:57:28.119189 3925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115: Neither blocks nor log segments found. Creating new log.
I20260925 22:57:28.120065 3925 log.cc:824] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115: Log is configured to *not* fsync() on all Append() calls
I20260925 22:57:28.122478 3925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115: No bootstrap required, opened a new log
Sep 25 22:57:28 dist-test-slave-x1lh krb5kdc[3844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:57:28.126521 3925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0351ec2f6f864dcfa85800eb53556115" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37499 } }
I20260925 22:57:28.126813 3925 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:57:28.126874 3925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0351ec2f6f864dcfa85800eb53556115, State: Initialized, Role: FOLLOWER
I20260925 22:57:28.127345 3925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0351ec2f6f864dcfa85800eb53556115" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37499 } }
I20260925 22:57:28.127521 3925 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:57:28.127611 3925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:57:28.127735 3925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:57:28.128736 3925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0351ec2f6f864dcfa85800eb53556115" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37499 } }
I20260925 22:57:28.129151 3925 leader_election.cc:302] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0351ec2f6f864dcfa85800eb53556115; no voters:
I20260925 22:57:28.129513 3925 leader_election.cc:288] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:57:28.129668 3930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:57:28.129921 3930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 1 LEADER]: Becoming Leader. State: Replica: 0351ec2f6f864dcfa85800eb53556115, State: Running, Role: LEADER
I20260925 22:57:28.130362 3930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0351ec2f6f864dcfa85800eb53556115" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37499 } }
I20260925 22:57:28.130913 3925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:57:28.131774 3930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0351ec2f6f864dcfa85800eb53556115. Latest consensus state: current_term: 1 leader_uuid: "0351ec2f6f864dcfa85800eb53556115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0351ec2f6f864dcfa85800eb53556115" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37499 } } }
I20260925 22:57:28.132014 3930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [sys.catalog]: This master's current role is: LEADER
I20260925 22:57:28.132223 3931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0351ec2f6f864dcfa85800eb53556115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0351ec2f6f864dcfa85800eb53556115" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37499 } } }
I20260925 22:57:28.132344 3931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [sys.catalog]: This master's current role is: LEADER
I20260925 22:57:28.132220 3928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:28.116628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43458 (local address 127.20.114.254:37499)
0925 22:57:28.117149 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:28.117165 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:28.117204 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:57:28.117925 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:28.117945 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:28.118314 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:28.118620 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:28.119642 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.120803 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.122048 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.122672 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.122868 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:28.125892 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:28.125899 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:28.125917 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:28.125951 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:28.128116 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:28.128648 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:28.128654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:28.128659 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:28.128781 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:28.129124 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:28.129132 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:28.129133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:28.129726 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:28.130634 (+ 908us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:28.130792 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 22:57:28.131115 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":184,"threads_started":1}
I20260925 22:57:28.132474 3939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:57:28.135776 3939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:57:28.143118 3939 catalog_manager.cc:1347] Generated new cluster ID: 2282e0f3236a45b5b943ed5041f35782
I20260925 22:57:28.143219 3939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:57:28.167783 3939 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:57:28.168957 3939 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:57:28.175771 3939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115: Generated new TSK 0
I20260925 22:57:28.176595 3939 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.114.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
I20260925 22:57:28.212656 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37499
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:57:28.337821 3951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:28.338182 3951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:28.338263 3951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:28.342614 3951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:28.342734 3951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:28.342833 3951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:57:28.348565 3951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:28.350093 3951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:28.351250 3951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:57:28.361455 3959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:28.361447 3956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:28.361479 3957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:28.362058 3951 server_base.cc:1034] running on GCE node
I20260925 22:57:28.362511 3951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:28.363198 3951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:28.364367 3951 hybrid_clock.cc:648] HybridClock initialized: now 1790377048364341 us; error 40 us; skew 500 ppm
Sep 25 22:57:28 dist-test-slave-x1lh krb5kdc[3844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377048, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:28.367465 3951 init.cc:377] Logged in from keytab as kudu/127.20.114.193@KRBTEST.COM (short username kudu)
I20260925 22:57:28.368520 3951 webserver.cc:460] Webserver started at http://127.20.114.193:44881/ using document root <none> and password file <none>
I20260925 22:57:28.369231 3951 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:28.369326 3951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:28.369580 3951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:57:28.371765 3951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "622a71eba47c4aaeac3c04f2c3eb199c"
format_stamp: "Formatted at 2026-09-25 22:57:28 on dist-test-slave-x1lh"
server_key: "95cf380f1a587840b5ecb456b234d549"
server_key_iv: "7823b1059c7d2134640063d732c52d94"
server_key_version: "encryptionkey@0"
I20260925 22:57:28.372401 3951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "622a71eba47c4aaeac3c04f2c3eb199c"
format_stamp: "Formatted at 2026-09-25 22:57:28 on dist-test-slave-x1lh"
server_key: "95cf380f1a587840b5ecb456b234d549"
server_key_iv: "7823b1059c7d2134640063d732c52d94"
server_key_version: "encryptionkey@0"
I20260925 22:57:28.376396 3951 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:57:28.379148 3966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:28.380467 3951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:57:28.380621 3951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "622a71eba47c4aaeac3c04f2c3eb199c"
format_stamp: "Formatted at 2026-09-25 22:57:28 on dist-test-slave-x1lh"
server_key: "95cf380f1a587840b5ecb456b234d549"
server_key_iv: "7823b1059c7d2134640063d732c52d94"
server_key_version: "encryptionkey@0"
I20260925 22:57:28.380749 3951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:28.392987 3951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:28.396515 3951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:28.396934 3951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:28.397634 3951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:57:28.398566 3951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:57:28.398618 3951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:28.398693 3951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:57:28.398728 3951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:28.408821 3951 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:36535
I20260925 22:57:28.408951 4079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:36535 every 8 connection(s)
I20260925 22:57:28.409993 3951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:57:28.410282 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 3951
I20260925 22:57:28.410430 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:57:28.410808 20939 external_mini_cluster.cc:1428] Setting key bfe512253072526a9fc69e7c981eff63
WARNING: no policy specified for kudu/127.20.114.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.194@KRBTEST.COM" created.
Sep 25 22:57:28 dist-test-slave-x1lh krb5kdc[3844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377048, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:57:28.431593 3928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:28.412353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:56799 (local address 127.20.114.254:37499)
0925 22:57:28.412496 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:28.412501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:28.413816 (+ 1315us) server_negotiation.cc:407] Connection header received
0925 22:57:28.415860 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:28.415863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:28.415979 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:28.416063 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:28.417626 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.418694 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.420114 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.420576 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.420721 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:28.426754 (+ 6033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:28.426758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:28.426760 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:28.426794 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:28.428504 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:28.429381 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:28.429384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:28.429385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:28.429505 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:28.430012 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:28.430014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:28.430015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:28.430215 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:28.430297 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:28.431203 (+ 906us) server_negotiation.cc:299] Negotiation successful
0925 22:57:28.431464 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:57:28.433068 4082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:28.413060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.193:56799)
0925 22:57:28.413653 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:28.413704 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:28.415514 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:28.416326 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:28.416335 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:28.416770 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:28.417410 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:28.417428 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.418891 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.418894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:28.419953 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:28.419965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.420773 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.420776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:28.421020 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:28.421730 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:28.421749 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:28.426460 (+ 4711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:28.428671 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:28.428682 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:28.428706 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:28.429195 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:28.429617 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:28.429624 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:28.429630 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:28.429900 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:28.430346 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:28.430358 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:28.430857 (+ 499us) client_negotiation.cc:769] Sending connection context
0925 22:57:28.431116 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 22:57:28.431628 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":142,"threads_started":1}
I20260925 22:57:28.434595 4080 heartbeater.cc:347] Connected to a master server at 127.20.114.254:37499
I20260925 22:57:28.434875 4080 heartbeater.cc:464] Registering TS with master...
I20260925 22:57:28.435422 4080 heartbeater.cc:511] Master 127.20.114.254:37499 requested a full tablet report, sending...
I20260925 22:57:28.437182 3889 ts_manager.cc:194] Registered new tserver with Master: 622a71eba47c4aaeac3c04f2c3eb199c (127.20.114.193:36535)
I20260925 22:57:28.438532 3889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:56799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
I20260925 22:57:28.445267 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37499
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:57:28.564649 4085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:28.565076 4085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:28.565163 4085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:28.569527 4085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:28.569624 4085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:28.569720 4085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:57:28.575438 4085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:28.576663 4085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:28.577872 4085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:57:28.588883 4093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:28.588846 4091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:28.588851 4090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:28.589340 4085 server_base.cc:1034] running on GCE node
I20260925 22:57:28.589800 4085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:28.590545 4085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:28.591748 4085 hybrid_clock.cc:648] HybridClock initialized: now 1790377048591681 us; error 83 us; skew 500 ppm
Sep 25 22:57:28 dist-test-slave-x1lh krb5kdc[3844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377048, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:28.594914 4085 init.cc:377] Logged in from keytab as kudu/127.20.114.194@KRBTEST.COM (short username kudu)
I20260925 22:57:28.595952 4085 webserver.cc:460] Webserver started at http://127.20.114.194:34483/ using document root <none> and password file <none>
I20260925 22:57:28.596550 4085 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:28.596639 4085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:28.596901 4085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:57:28.599063 4085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "7e931ecace514df79840014f6ecb2038"
format_stamp: "Formatted at 2026-09-25 22:57:28 on dist-test-slave-x1lh"
server_key: "adbb15d9f9127ed576ea325d9f4e419b"
server_key_iv: "4a96185726dbecaa2aa490ebc6a3c06e"
server_key_version: "encryptionkey@0"
I20260925 22:57:28.599742 4085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "7e931ecace514df79840014f6ecb2038"
format_stamp: "Formatted at 2026-09-25 22:57:28 on dist-test-slave-x1lh"
server_key: "adbb15d9f9127ed576ea325d9f4e419b"
server_key_iv: "4a96185726dbecaa2aa490ebc6a3c06e"
server_key_version: "encryptionkey@0"
I20260925 22:57:28.604050 4085 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 22:57:28.607179 4100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:28.608651 4085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 22:57:28.608824 4085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "7e931ecace514df79840014f6ecb2038"
format_stamp: "Formatted at 2026-09-25 22:57:28 on dist-test-slave-x1lh"
server_key: "adbb15d9f9127ed576ea325d9f4e419b"
server_key_iv: "4a96185726dbecaa2aa490ebc6a3c06e"
server_key_version: "encryptionkey@0"
I20260925 22:57:28.609038 4085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:28.622742 4085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:28.626542 4085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:28.626760 4085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:28.627414 4085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:57:28.628319 4085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:57:28.628391 4085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:28.628461 4085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:57:28.628513 4085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:28.638190 4085 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:44883
I20260925 22:57:28.638216 4213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:44883 every 8 connection(s)
I20260925 22:57:28.639138 4085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:57:28.642297 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 4085
I20260925 22:57:28.642413 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:57:28.642736 20939 external_mini_cluster.cc:1428] Setting key 87913ff3d33854ff5cc01877b5646bb1
Sep 25 22:57:28 dist-test-slave-x1lh krb5kdc[3844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377048, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.114.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.195@KRBTEST.COM" created.
I20260925 22:57:28.656522 3928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:28.641358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:45251 (local address 127.20.114.254:37499)
0925 22:57:28.641494 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:28.641497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:28.642331 (+ 834us) server_negotiation.cc:407] Connection header received
0925 22:57:28.644061 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:28.644065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:28.644118 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:28.644222 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:28.646105 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.646691 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.648551 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.648804 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.649020 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:28.652296 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:28.652300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:28.652302 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:28.652331 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:28.654015 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:28.654674 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:28.654678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:28.654679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:28.654768 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:28.655146 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:28.655148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:28.655149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:28.655339 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:28.655418 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:28.656088 (+ 670us) server_negotiation.cc:299] Negotiation successful
0925 22:57:28.656288 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:57:28.657548 4216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:28.641599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.194:45251)
0925 22:57:28.642129 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:28.642194 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:28.643818 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:28.644565 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:28.644575 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:28.645167 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:28.645921 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:28.645945 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.646835 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.646841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:28.648394 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:28.648410 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.649088 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.649096 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:28.649365 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:28.650099 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:28.650128 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:28.652071 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:28.654161 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:28.654168 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:28.654191 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:28.654531 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:28.654881 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:28.654883 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:28.654885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:28.655048 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:28.655457 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:28.655464 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:28.655828 (+ 364us) client_negotiation.cc:769] Sending connection context
0925 22:57:28.656027 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 22:57:28.656384 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":119,"threads_started":1}
I20260925 22:57:28.658878 4214 heartbeater.cc:347] Connected to a master server at 127.20.114.254:37499
I20260925 22:57:28.659188 4214 heartbeater.cc:464] Registering TS with master...
I20260925 22:57:28.659811 4214 heartbeater.cc:511] Master 127.20.114.254:37499 requested a full tablet report, sending...
I20260925 22:57:28.661082 3889 ts_manager.cc:194] Registered new tserver with Master: 7e931ecace514df79840014f6ecb2038 (127.20.114.194:44883)
I20260925 22:57:28.661739 3889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:45251
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
I20260925 22:57:28.676556 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37499
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:57:28.792171 4219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:28.792487 4219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:28.792533 4219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:28.796661 4219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:28.796733 4219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:28.796797 4219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:57:28.802505 4219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:37499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:28.803610 4219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:28.804587 4219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:57:28.814821 4227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:28.814823 4224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:28.814821 4225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:28.815622 4219 server_base.cc:1034] running on GCE node
I20260925 22:57:28.816223 4219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:28.817042 4219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:28.818260 4219 hybrid_clock.cc:648] HybridClock initialized: now 1790377048818216 us; error 58 us; skew 500 ppm
Sep 25 22:57:28 dist-test-slave-x1lh krb5kdc[3844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377048, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:28.821442 4219 init.cc:377] Logged in from keytab as kudu/127.20.114.195@KRBTEST.COM (short username kudu)
I20260925 22:57:28.822355 4219 webserver.cc:460] Webserver started at http://127.20.114.195:41493/ using document root <none> and password file <none>
I20260925 22:57:28.822971 4219 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:28.823042 4219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:28.823217 4219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:57:28.825565 4219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "547ccf22dc044da08b3b8636fd64a2ab"
format_stamp: "Formatted at 2026-09-25 22:57:28 on dist-test-slave-x1lh"
server_key: "f9e23b273bbc3a797015ec0c12d6e599"
server_key_iv: "086b5481dce447e4fce9d144016016a2"
server_key_version: "encryptionkey@0"
I20260925 22:57:28.826170 4219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "547ccf22dc044da08b3b8636fd64a2ab"
format_stamp: "Formatted at 2026-09-25 22:57:28 on dist-test-slave-x1lh"
server_key: "f9e23b273bbc3a797015ec0c12d6e599"
server_key_iv: "086b5481dce447e4fce9d144016016a2"
server_key_version: "encryptionkey@0"
I20260925 22:57:28.830288 4219 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:57:28.834365 4234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:28.835768 4219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:57:28.835922 4219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "547ccf22dc044da08b3b8636fd64a2ab"
format_stamp: "Formatted at 2026-09-25 22:57:28 on dist-test-slave-x1lh"
server_key: "f9e23b273bbc3a797015ec0c12d6e599"
server_key_iv: "086b5481dce447e4fce9d144016016a2"
server_key_version: "encryptionkey@0"
I20260925 22:57:28.836071 4219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:28.865949 4219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:28.869356 4219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:28.869568 4219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:28.870241 4219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:57:28.871127 4219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:57:28.871174 4219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:28.871240 4219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:57:28.871273 4219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:28.880649 4219 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:33565
I20260925 22:57:28.880673 4347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:33565 every 8 connection(s)
I20260925 22:57:28.881799 4219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:57:28.883948 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 4219
I20260925 22:57:28.884141 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:57:28.884471 20939 external_mini_cluster.cc:1428] Setting key d3c8110d119610535a3fc62638fccfb3
Sep 25 22:57:28 dist-test-slave-x1lh krb5kdc[3844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377048, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:57:28.897130 3928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:28.883851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:55261 (local address 127.20.114.254:37499)
0925 22:57:28.884006 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:28.884011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:28.885392 (+ 1381us) server_negotiation.cc:407] Connection header received
0925 22:57:28.886823 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:28.886827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:28.886885 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:28.886964 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:28.888229 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.888770 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.890065 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.890287 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.890400 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:28.893062 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:28.893065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:28.893067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:28.893096 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:28.894821 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:28.895447 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:28.895451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:28.895453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:28.895546 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:28.895918 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:28.895921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:28.895922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:28.896130 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:28.896207 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:28.896770 (+ 563us) server_negotiation.cc:299] Negotiation successful
0925 22:57:28.896968 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:57:28.898072 4350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:28.884453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.195:55261)
0925 22:57:28.885229 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:28.885279 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:28.886600 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:28.887161 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:28.887169 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:28.887562 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:28.888076 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:28.888090 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.888934 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.888938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:28.889921 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:28.889931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.890447 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.890451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:28.890641 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:28.891247 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:28.891265 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:28.892792 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:28.894975 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:28.894982 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:28.894999 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:28.895307 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:28.895645 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:28.895647 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:28.895649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:28.895805 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:28.896221 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:28.896227 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:28.896491 (+ 264us) client_negotiation.cc:769] Sending connection context
0925 22:57:28.896693 (+ 202us) client_negotiation.cc:240] Negotiation successful
0925 22:57:28.897044 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":138,"threads_started":1}
I20260925 22:57:28.899366 4348 heartbeater.cc:347] Connected to a master server at 127.20.114.254:37499
I20260925 22:57:28.899592 4348 heartbeater.cc:464] Registering TS with master...
I20260925 22:57:28.900127 4348 heartbeater.cc:511] Master 127.20.114.254:37499 requested a full tablet report, sending...
I20260925 22:57:28.901278 3888 ts_manager.cc:194] Registered new tserver with Master: 547ccf22dc044da08b3b8636fd64a2ab (127.20.114.195:33565)
I20260925 22:57:28.901862 3888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:55261
I20260925 22:57:28.911607 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:57:28.921299 3928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:28.913706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43472 (local address 127.20.114.254:37499)
0925 22:57:28.913859 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:28.913865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:28.914111 (+ 246us) server_negotiation.cc:407] Connection header received
0925 22:57:28.914207 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:28.914209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:28.914253 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:28.914321 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:28.915027 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.915544 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.916386 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.916627 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.916710 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:28.918131 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:28.918134 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:28.918136 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:28.918161 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:28.919620 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:28.920024 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:28.920027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:28.920028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:28.920104 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:28.920333 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:28.920335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:28.920336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:28.920583 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:28.920689 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:28.921005 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 22:57:28.921107 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:57:28.924698 3888 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:57:28.927920 3888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:57:28.943969 4361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:28.937599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34334 (local address 127.20.114.194:44883)
0925 22:57:28.938121 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:28.938133 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:28.938162 (+ 29us) server_negotiation.cc:407] Connection header received
0925 22:57:28.938385 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:28.938391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:28.938625 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:28.938784 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:57:28.939360 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.940416 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.942468 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.943168 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.943317 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:28.943467 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:28.943673 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 22:57:28.943782 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":151,"threads_started":1}
I20260925 22:57:28.943964 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:28.937418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:44883 (local address 127.0.0.1:34334)
0925 22:57:28.937985 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:28.938043 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:28.938167 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:28.939005 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:28.939011 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:28.939039 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:57:28.939200 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:28.939208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.940551 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.940555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:28.942252 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:28.942263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.943308 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.943312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:28.943449 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:28.943499 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 22:57:28.943639 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 22:57:28.943782 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":44,"thread_start_us":100,"threads_started":1}
I20260925 22:57:28.944201 4359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:28.938141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:33565 (local address 127.0.0.1:55310)
0925 22:57:28.939129 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:28.939142 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:28.939232 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:28.939633 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:28.939636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:28.939652 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:57:28.939724 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:28.939729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.941069 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.941073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:28.942342 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:28.942424 (+ 82us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.943667 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.943671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:28.943819 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:28.943891 (+ 72us) client_negotiation.cc:769] Sending connection context
0925 22:57:28.943965 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 22:57:28.944061 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"mutex_wait_us":166,"thread_start_us":95,"threads_started":1}
I20260925 22:57:28.944306 4363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:28.938226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52162 (local address 127.20.114.193:36535)
0925 22:57:28.938926 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:28.938934 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:28.938961 (+ 27us) server_negotiation.cc:407] Connection header received
0925 22:57:28.939036 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:28.939043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:28.939268 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:28.939414 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:57:28.939611 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.940805 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.942474 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.943326 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.943463 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:28.943592 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:28.944049 (+ 457us) server_negotiation.cc:299] Negotiation successful
0925 22:57:28.944151 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":69,"threads_started":1}
I20260925 22:57:28.944700 4364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:28.938552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55310 (local address 127.20.114.195:33565)
0925 22:57:28.939011 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:28.939022 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:28.939244 (+ 222us) server_negotiation.cc:407] Connection header received
0925 22:57:28.939333 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:28.939340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:28.939511 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:28.939653 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:57:28.939799 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.940930 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.942531 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.943326 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.943463 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:28.943592 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:28.944396 (+ 804us) server_negotiation.cc:299] Negotiation successful
0925 22:57:28.944550 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":161,"threads_started":1}
I20260925 22:57:28.945077 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:28.937611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:36535 (local address 127.0.0.1:52162)
0925 22:57:28.938583 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:28.938596 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:28.938686 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:28.939387 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:28.939392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:28.939410 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:57:28.939489 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:28.939494 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.940937 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.940940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:28.942362 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:28.942374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:28.943660 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:28.943664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:28.943850 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:28.943885 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 22:57:28.943966 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 22:57:28.944953 (+ 987us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":1067,"thread_start_us":99,"threads_started":1}
I20260925 22:57:28.947420 4148 tablet_service.cc:1467] Processing CreateTablet for tablet f15bc7aaea87443db116746273ff1310 (DEFAULT_TABLE table=test-table [id=aef98bd7989841fca1d06fbc552e109f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:57:28.947578 4282 tablet_service.cc:1467] Processing CreateTablet for tablet f15bc7aaea87443db116746273ff1310 (DEFAULT_TABLE table=test-table [id=aef98bd7989841fca1d06fbc552e109f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:57:28.948508 4148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15bc7aaea87443db116746273ff1310. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:57:28.948832 4282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15bc7aaea87443db116746273ff1310. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:57:28.951802 4014 tablet_service.cc:1467] Processing CreateTablet for tablet f15bc7aaea87443db116746273ff1310 (DEFAULT_TABLE table=test-table [id=aef98bd7989841fca1d06fbc552e109f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:57:28.952804 4014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15bc7aaea87443db116746273ff1310. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:57:28.955971 4365 tablet_bootstrap.cc:492] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab: Bootstrap starting.
I20260925 22:57:28.958722 4366 tablet_bootstrap.cc:492] T f15bc7aaea87443db116746273ff1310 P 7e931ecace514df79840014f6ecb2038: Bootstrap starting.
I20260925 22:57:28.958992 4365 tablet_bootstrap.cc:654] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab: Neither blocks nor log segments found. Creating new log.
I20260925 22:57:28.959985 4367 tablet_bootstrap.cc:492] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c: Bootstrap starting.
I20260925 22:57:28.959946 4365 log.cc:824] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab: Log is configured to *not* fsync() on all Append() calls
I20260925 22:57:28.961607 4366 tablet_bootstrap.cc:654] T f15bc7aaea87443db116746273ff1310 P 7e931ecace514df79840014f6ecb2038: Neither blocks nor log segments found. Creating new log.
I20260925 22:57:28.962383 4367 tablet_bootstrap.cc:654] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c: Neither blocks nor log segments found. Creating new log.
I20260925 22:57:28.962460 4366 log.cc:824] T f15bc7aaea87443db116746273ff1310 P 7e931ecace514df79840014f6ecb2038: Log is configured to *not* fsync() on all Append() calls
I20260925 22:57:28.962745 4365 tablet_bootstrap.cc:492] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab: No bootstrap required, opened a new log
I20260925 22:57:28.962980 4365 ts_tablet_manager.cc:1397] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260925 22:57:28.963212 4367 log.cc:824] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:57:28.964526 4366 tablet_bootstrap.cc:492] T f15bc7aaea87443db116746273ff1310 P 7e931ecace514df79840014f6ecb2038: No bootstrap required, opened a new log
I20260925 22:57:28.964732 4366 ts_tablet_manager.cc:1397] T f15bc7aaea87443db116746273ff1310 P 7e931ecace514df79840014f6ecb2038: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:57:28.965233 4367 tablet_bootstrap.cc:492] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c: No bootstrap required, opened a new log
I20260925 22:57:28.965408 4367 ts_tablet_manager.cc:1397] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:57:28.966384 4365 raft_consensus.cc:348] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } }
I20260925 22:57:28.966646 4365 raft_consensus.cc:374] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:57:28.966727 4365 raft_consensus.cc:729] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547ccf22dc044da08b3b8636fd64a2ab, State: Initialized, Role: FOLLOWER
I20260925 22:57:28.967228 4365 consensus_queue.cc:260] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } }
I20260925 22:57:28.968613 4365 ts_tablet_manager.cc:1428] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260925 22:57:28.969014 4348 heartbeater.cc:503] Master 127.20.114.254:37499 was elected leader, sending a full tablet report...
I20260925 22:57:28.968907 4366 raft_consensus.cc:348] T f15bc7aaea87443db116746273ff1310 P 7e931ecace514df79840014f6ecb2038 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } }
I20260925 22:57:28.969228 4366 raft_consensus.cc:374] T f15bc7aaea87443db116746273ff1310 P 7e931ecace514df79840014f6ecb2038 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:57:28.969287 4366 raft_consensus.cc:729] T f15bc7aaea87443db116746273ff1310 P 7e931ecace514df79840014f6ecb2038 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e931ecace514df79840014f6ecb2038, State: Initialized, Role: FOLLOWER
I20260925 22:57:28.969750 4366 consensus_queue.cc:260] T f15bc7aaea87443db116746273ff1310 P 7e931ecace514df79840014f6ecb2038 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } }
I20260925 22:57:28.969869 4367 raft_consensus.cc:348] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } }
I20260925 22:57:28.970146 4367 raft_consensus.cc:374] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:57:28.970204 4367 raft_consensus.cc:729] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622a71eba47c4aaeac3c04f2c3eb199c, State: Initialized, Role: FOLLOWER
I20260925 22:57:28.970644 4367 consensus_queue.cc:260] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } }
I20260925 22:57:28.970829 4366 ts_tablet_manager.cc:1428] T f15bc7aaea87443db116746273ff1310 P 7e931ecace514df79840014f6ecb2038: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260925 22:57:28.971181 4214 heartbeater.cc:503] Master 127.20.114.254:37499 was elected leader, sending a full tablet report...
I20260925 22:57:28.971275 4080 heartbeater.cc:503] Master 127.20.114.254:37499 was elected leader, sending a full tablet report...
I20260925 22:57:28.971606 4367 ts_tablet_manager.cc:1428] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:57:29.017516 4371 raft_consensus.cc:482] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:57:29.017719 4371 raft_consensus.cc:504] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } }
I20260925 22:57:29.018930 4371 leader_election.cc:288] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e931ecace514df79840014f6ecb2038 (127.20.114.194:44883), 622a71eba47c4aaeac3c04f2c3eb199c (127.20.114.193:36535)
I20260925 22:57:29.022565 4361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:29.019219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:55851 (local address 127.20.114.194:44883)
0925 22:57:29.019403 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:29.019408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:29.019426 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:57:29.019599 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:29.019601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:29.019646 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:29.019725 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:57:29.020009 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:29.020494 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:29.021584 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:29.022006 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:29.022135 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:29.022187 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:29.022362 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 22:57:29.022428 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 22:57:29.022557 4350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:29.019125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:44883 (local address 127.20.114.195:55851)
0925 22:57:29.019316 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:29.019333 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:29.019439 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:29.019748 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:29.019751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:29.019767 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:57:29.019889 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:29.019894 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:29.020602 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:29.020607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:29.021456 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:29.021465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:29.022128 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:29.022132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:29.022240 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:29.022258 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 22:57:29.022345 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 22:57:29.022419 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260925 22:57:29.022703 4363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:29.019393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:38859 (local address 127.20.114.193:36535)
0925 22:57:29.019542 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:29.019548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:29.019676 (+ 128us) server_negotiation.cc:407] Connection header received
0925 22:57:29.019828 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:29.019832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:29.019871 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:29.019954 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:57:29.020211 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:29.020721 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:29.021751 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:29.022171 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:29.022301 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:29.022353 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:29.022521 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 22:57:29.022593 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:57:29.022716 4374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:29.019181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:36535 (local address 127.20.114.195:38859)
0925 22:57:29.019593 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:29.019603 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:29.019717 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:29.019961 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:29.019964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:29.019979 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:57:29.020106 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:29.020111 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:29.020834 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:29.020839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:29.021627 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:29.021635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:29.022290 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:29.022294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:29.022391 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:29.022420 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 22:57:29.022503 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 22:57:29.022573 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":123,"threads_started":1}
I20260925 22:57:29.023345 4168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15bc7aaea87443db116746273ff1310" candidate_uuid: "547ccf22dc044da08b3b8636fd64a2ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e931ecace514df79840014f6ecb2038" is_pre_election: true
I20260925 22:57:29.023342 4034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15bc7aaea87443db116746273ff1310" candidate_uuid: "547ccf22dc044da08b3b8636fd64a2ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" is_pre_election: true
I20260925 22:57:29.023658 4168 raft_consensus.cc:2454] T f15bc7aaea87443db116746273ff1310 P 7e931ecace514df79840014f6ecb2038 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 547ccf22dc044da08b3b8636fd64a2ab in term 0.
I20260925 22:57:29.023671 4034 raft_consensus.cc:2454] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 547ccf22dc044da08b3b8636fd64a2ab in term 0.
I20260925 22:57:29.024221 4238 leader_election.cc:302] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 547ccf22dc044da08b3b8636fd64a2ab, 622a71eba47c4aaeac3c04f2c3eb199c, 7e931ecace514df79840014f6ecb2038; no voters:
I20260925 22:57:29.024490 4371 raft_consensus.cc:2781] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:57:29.024587 4371 raft_consensus.cc:482] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:57:29.024649 4371 raft_consensus.cc:3037] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:57:29.025727 4371 raft_consensus.cc:504] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } }
I20260925 22:57:29.026139 4371 leader_election.cc:288] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab [CANDIDATE]: Term 1 election: Requested vote from peers 7e931ecace514df79840014f6ecb2038 (127.20.114.194:44883), 622a71eba47c4aaeac3c04f2c3eb199c (127.20.114.193:36535)
I20260925 22:57:29.026494 4168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15bc7aaea87443db116746273ff1310" candidate_uuid: "547ccf22dc044da08b3b8636fd64a2ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e931ecace514df79840014f6ecb2038"
I20260925 22:57:29.026515 4034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15bc7aaea87443db116746273ff1310" candidate_uuid: "547ccf22dc044da08b3b8636fd64a2ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622a71eba47c4aaeac3c04f2c3eb199c"
I20260925 22:57:29.026633 4168 raft_consensus.cc:3037] T f15bc7aaea87443db116746273ff1310 P 7e931ecace514df79840014f6ecb2038 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:57:29.026644 4034 raft_consensus.cc:3037] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:57:29.027676 4034 raft_consensus.cc:2454] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 547ccf22dc044da08b3b8636fd64a2ab in term 1.
I20260925 22:57:29.027680 4168 raft_consensus.cc:2454] T f15bc7aaea87443db116746273ff1310 P 7e931ecace514df79840014f6ecb2038 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 547ccf22dc044da08b3b8636fd64a2ab in term 1.
I20260925 22:57:29.028049 4238 leader_election.cc:302] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 547ccf22dc044da08b3b8636fd64a2ab, 622a71eba47c4aaeac3c04f2c3eb199c; no voters:
I20260925 22:57:29.028213 4371 raft_consensus.cc:2781] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:57:29.028426 4371 raft_consensus.cc:686] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab [term 1 LEADER]: Becoming Leader. State: Replica: 547ccf22dc044da08b3b8636fd64a2ab, State: Running, Role: LEADER
I20260925 22:57:29.028770 4371 consensus_queue.cc:237] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } }
I20260925 22:57:29.032161 3887 catalog_manager.cc:5495] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 547ccf22dc044da08b3b8636fd64a2ab (127.20.114.195). New cstate: current_term: 1 leader_uuid: "547ccf22dc044da08b3b8636fd64a2ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } health_report { overall_health: HEALTHY } } }
I20260925 22:57:29.044402 4364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:29.040559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55312 (local address 127.20.114.195:33565)
0925 22:57:29.040693 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:29.040697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:29.040713 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:57:29.040782 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:29.040784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:29.040838 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:29.041009 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:57:29.041312 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:29.041818 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:29.042851 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:29.043110 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:29.043196 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:29.043522 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:57:29.044011 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:57:29.044108 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:29.044214 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 22:57:29.044267 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:57:29.050611 4168 raft_consensus.cc:1261] T f15bc7aaea87443db116746273ff1310 P 7e931ecace514df79840014f6ecb2038 [term 1 FOLLOWER]: Refusing update from remote peer 547ccf22dc044da08b3b8636fd64a2ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:57:29.050611 4034 raft_consensus.cc:1261] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c [term 1 FOLLOWER]: Refusing update from remote peer 547ccf22dc044da08b3b8636fd64a2ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:57:29.051359 4371 consensus_queue.cc:1035] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:57:29.051570 4375 consensus_queue.cc:1035] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:57:29.058689 4378 mvcc.cc:205] Tried to move back new op lower bound from 7333384392903319552 to 7333384392824758272. Current Snapshot: MvccSnapshot[applied={T|T < 7333384392903319552}]
I20260925 22:57:29.060102 4379 mvcc.cc:205] Tried to move back new op lower bound from 7333384392903319552 to 7333384392824758272. Current Snapshot: MvccSnapshot[applied={T|T < 7333384392903319552}]
I20260925 22:57:29.067641 3887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:57:29.067801 3887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:57:29.070187 3887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115: Sending DeleteTablet for 3 replicas of tablet f15bc7aaea87443db116746273ff1310
I20260925 22:57:29.071059 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 3951
I20260925 22:57:29.071020 4014 tablet_service.cc:1514] Processing DeleteTablet for tablet f15bc7aaea87443db116746273ff1310 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:57:29 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:52162
I20260925 22:57:29.071251 4148 tablet_service.cc:1514] Processing DeleteTablet for tablet f15bc7aaea87443db116746273ff1310 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:57:29 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:34334
I20260925 22:57:29.071655 4282 tablet_service.cc:1514] Processing DeleteTablet for tablet f15bc7aaea87443db116746273ff1310 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:57:29 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:55310
I20260925 22:57:29.072173 4388 tablet_replica.cc:330] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab: stopping tablet replica
I20260925 22:57:29.072482 4388 raft_consensus.cc:2229] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:57:29.072956 4388 raft_consensus.cc:2258] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:57:29.073391 4387 tablet_replica.cc:330] T f15bc7aaea87443db116746273ff1310 P 7e931ecace514df79840014f6ecb2038: stopping tablet replica
I20260925 22:57:29.073685 4387 raft_consensus.cc:2229] T f15bc7aaea87443db116746273ff1310 P 7e931ecace514df79840014f6ecb2038 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:57:29.073992 4387 raft_consensus.cc:2258] T f15bc7aaea87443db116746273ff1310 P 7e931ecace514df79840014f6ecb2038 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:57:29.075600 4388 ts_tablet_manager.cc:1905] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:57:29.075600 4387 ts_tablet_manager.cc:1905] T f15bc7aaea87443db116746273ff1310 P 7e931ecace514df79840014f6ecb2038: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:57:29.078447 4388 ts_tablet_manager.cc:1918] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:57:29.078447 4387 ts_tablet_manager.cc:1918] T f15bc7aaea87443db116746273ff1310 P 7e931ecace514df79840014f6ecb2038: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:57:29.078559 4387 log.cc:1196] T f15bc7aaea87443db116746273ff1310 P 7e931ecace514df79840014f6ecb2038: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/f15bc7aaea87443db116746273ff1310
I20260925 22:57:29.078681 4388 log.cc:1196] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/f15bc7aaea87443db116746273ff1310
W20260925 22:57:29.078655 3876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.114.193:36535 (error 108)
I20260925 22:57:29.078917 4387 ts_tablet_manager.cc:1939] T f15bc7aaea87443db116746273ff1310 P 7e931ecace514df79840014f6ecb2038: Deleting consensus metadata
I20260925 22:57:29.079118 4388 ts_tablet_manager.cc:1939] T f15bc7aaea87443db116746273ff1310 P 547ccf22dc044da08b3b8636fd64a2ab: Deleting consensus metadata
I20260925 22:57:29.079517 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 4085
I20260925 22:57:29.080401 3873 catalog_manager.cc:4841] TS 547ccf22dc044da08b3b8636fd64a2ab (127.20.114.195:33565): tablet f15bc7aaea87443db116746273ff1310 (table test-table [id=aef98bd7989841fca1d06fbc552e109f]) successfully deleted
I20260925 22:57:29.083086 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:29.081038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:36535 (local address 127.0.0.1:52166)
0925 22:57:29.082891 (+ 1853us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:29.082989 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.193:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1749}
W20260925 22:57:29.083290 3876 catalog_manager.cc:4584] TS 622a71eba47c4aaeac3c04f2c3eb199c (127.20.114.193:36535): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f15bc7aaea87443db116746273ff1310: Network error: Client connection negotiation failed: client connection to 127.20.114.193:36535: connect: Connection refused (error 111)
I20260925 22:57:29.085922 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 4219
I20260925 22:57:29.086793 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:29.086201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:44883 (local address 127.0.0.1:34340)
0925 22:57:29.086676 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:29.086725 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.194:44883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401}
I20260925 22:57:29.092132 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 3857
I20260925 22:57:49.099184 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:37499
--webserver_interface=127.20.114.254
--webserver_port=40287
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:37499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:57:49.212142 4390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:49.212469 4390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:49.212569 4390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:49.216718 4390 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 22:57:49.216775 4390 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 22:57:49.216792 4390 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 22:57:49.216806 4390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:57:49.216825 4390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:49.216879 4390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:49.216943 4390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:57:49.216970 4390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:57:49.222510 4390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:37499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:37499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=40287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:49.223714 4390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:49.225032 4390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:57:49.238863 4399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:49.238857 4397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:49.238857 4396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:49.239156 4390 server_base.cc:1034] running on GCE node
I20260925 22:57:49.240049 4390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:49.241384 4390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:49.242588 4390 hybrid_clock.cc:648] HybridClock initialized: now 1790377069242573 us; error 48 us; skew 500 ppm
Sep 25 22:57:49 dist-test-slave-x1lh krb5kdc[3844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377069, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:49.250533 4390 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:57:49.251621 4390 webserver.cc:460] Webserver started at http://127.20.114.254:40287/ using document root <none> and password file <none>
I20260925 22:57:49.252272 4390 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:49.252396 4390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:49.257014 4390 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:57:49.259603 4406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:49.261180 4390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:57:49.261360 4390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "0351ec2f6f864dcfa85800eb53556115"
format_stamp: "Formatted at 2026-09-25 22:57:28 on dist-test-slave-x1lh"
server_key: "19120fcff5ef12ed91e7b80d854f1a31"
server_key_iv: "1efbb08296b3845f62b31883fcb9f014"
server_key_version: "encryptionkey@0"
I20260925 22:57:49.262060 4390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:49.297554 4390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:49.301044 4390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:49.301256 4390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:49.308745 4390 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:37499
I20260925 22:57:49.309122 4458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:37499 every 8 connection(s)
I20260925 22:57:49.309962 4390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:57:49.314634 4459 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:57:49.316412 4459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115: Bootstrap starting.
I20260925 22:57:49.316680 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 4390
I20260925 22:57:49.317251 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:36535
--local_ip_for_outbound_sockets=127.20.114.193
--tserver_master_addrs=127.20.114.254:37499
--webserver_port=44881
--webserver_interface=127.20.114.193
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:57:49.334334 4459 log.cc:824] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115: Log is configured to *not* fsync() on all Append() calls
I20260925 22:57:49.343439 4459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:57:49.343915 4459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115: Bootstrap complete.
I20260925 22:57:49.348198 4459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0351ec2f6f864dcfa85800eb53556115" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37499 } }
I20260925 22:57:49.348646 4459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0351ec2f6f864dcfa85800eb53556115, State: Initialized, Role: FOLLOWER
I20260925 22:57:49.349267 4459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0351ec2f6f864dcfa85800eb53556115" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37499 } }
I20260925 22:57:49.349417 4459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:57:49.349516 4459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:57:49.349618 4459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:57:49.351751 4459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0351ec2f6f864dcfa85800eb53556115" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37499 } }
I20260925 22:57:49.352118 4459 leader_election.cc:302] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0351ec2f6f864dcfa85800eb53556115; no voters:
I20260925 22:57:49.352437 4459 leader_election.cc:288] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:57:49.352612 4463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:57:49.352998 4463 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 2 LEADER]: Becoming Leader. State: Replica: 0351ec2f6f864dcfa85800eb53556115, State: Running, Role: LEADER
I20260925 22:57:49.353314 4463 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0351ec2f6f864dcfa85800eb53556115" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37499 } }
I20260925 22:57:49.353820 4459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:57:49.354903 4465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0351ec2f6f864dcfa85800eb53556115. Latest consensus state: current_term: 2 leader_uuid: "0351ec2f6f864dcfa85800eb53556115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0351ec2f6f864dcfa85800eb53556115" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37499 } } }
I20260925 22:57:49.354910 4464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0351ec2f6f864dcfa85800eb53556115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0351ec2f6f864dcfa85800eb53556115" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37499 } } }
I20260925 22:57:49.355039 4465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [sys.catalog]: This master's current role is: LEADER
I20260925 22:57:49.355044 4464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [sys.catalog]: This master's current role is: LEADER
I20260925 22:57:49.355396 4472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:57:49.360353 4472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:57:49.361079 4472 catalog_manager.cc:1259] Loaded cluster ID: 2282e0f3236a45b5b943ed5041f35782
I20260925 22:57:49.361171 4472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:57:49.363811 4472 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:57:49.364770 4472 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115: Loaded TSK: 0
I20260925 22:57:49.379897 4472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115: Generated new TSK 1
I20260925 22:57:49.380205 4472 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:57:49.443344 4461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:49.443667 4461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:49.443753 4461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:49.448010 4461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:49.448120 4461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:49.448215 4461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:57:49.453822 4461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:36535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=44881
--tserver_master_addrs=127.20.114.254:37499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:49.454932 4461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:49.456044 4461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:57:49.466706 4486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:49.466701 4487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:49.466935 4461 server_base.cc:1034] running on GCE node
W20260925 22:57:49.466715 4489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:49.467636 4461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:49.468344 4461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:49.469550 4461 hybrid_clock.cc:648] HybridClock initialized: now 1790377069469524 us; error 46 us; skew 500 ppm
Sep 25 22:57:49 dist-test-slave-x1lh krb5kdc[3844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377069, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:49.472545 4461 init.cc:377] Logged in from keytab as kudu/127.20.114.193@KRBTEST.COM (short username kudu)
I20260925 22:57:49.473721 4461 webserver.cc:460] Webserver started at http://127.20.114.193:44881/ using document root <none> and password file <none>
I20260925 22:57:49.474328 4461 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:49.474413 4461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:49.478823 4461 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 22:57:49.481462 4496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:49.483088 4461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:57:49.483271 4461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "622a71eba47c4aaeac3c04f2c3eb199c"
format_stamp: "Formatted at 2026-09-25 22:57:28 on dist-test-slave-x1lh"
server_key: "95cf380f1a587840b5ecb456b234d549"
server_key_iv: "7823b1059c7d2134640063d732c52d94"
server_key_version: "encryptionkey@0"
I20260925 22:57:49.483959 4461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:49.518306 4461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:49.522014 4461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:49.522236 4461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:49.522866 4461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:57:49.524045 4503 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:57:49.528398 4461 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:57:49.528505 4461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260925 22:57:49.528605 4461 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:57:49.530484 4461 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:57:49.530562 4461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260925 22:57:49.530907 4503 tablet_bootstrap.cc:492] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c: Bootstrap starting.
I20260925 22:57:49.541754 4461 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:36535
I20260925 22:57:49.541841 4610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:36535 every 8 connection(s)
I20260925 22:57:49.542829 4461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:57:49.544638 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 4461
I20260925 22:57:49.545266 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:44883
--local_ip_for_outbound_sockets=127.20.114.194
--tserver_master_addrs=127.20.114.254:37499
--webserver_port=34483
--webserver_interface=127.20.114.194
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:57:49.554397 4503 log.cc:824] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:57:49 dist-test-slave-x1lh krb5kdc[3844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377069, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:57:49.563763 4503 tablet_bootstrap.cc:492] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:57:49.564224 4503 tablet_bootstrap.cc:492] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c: Bootstrap complete.
I20260925 22:57:49.565761 4503 ts_tablet_manager.cc:1397] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c: Time spent bootstrapping tablet: real 0.035s user 0.023s sys 0.011s
I20260925 22:57:49.567255 4614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:49.545328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.193:58041)
0925 22:57:49.545980 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:49.546043 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:49.547366 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:49.550069 (+ 2703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:49.550082 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:49.550623 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:49.551402 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:49.551431 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:49.553609 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:49.553613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:49.555173 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:49.555183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:49.557582 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:49.557589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:49.557899 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:49.558635 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:49.558661 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:49.560698 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:49.563463 (+ 2765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:49.563470 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:49.563485 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:49.563911 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:49.564324 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:49.564327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:49.564329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:49.564503 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:49.565226 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:49.565236 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:49.565594 (+ 358us) client_negotiation.cc:769] Sending connection context
0925 22:57:49.565859 (+ 265us) client_negotiation.cc:240] Negotiation successful
0925 22:57:49.566221 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":148,"threads_started":1}
I20260925 22:57:49.569231 4613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:49.545367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:58041 (local address 127.20.114.254:37499)
0925 22:57:49.547955 (+ 2588us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:49.547979 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:49.548042 (+ 63us) server_negotiation.cc:407] Connection header received
0925 22:57:49.548407 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:49.548430 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:49.548962 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:49.549379 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:49.551671 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:49.553436 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:49.555409 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:49.557398 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:49.557577 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:49.560968 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:49.560975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:49.560993 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:49.561029 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:49.563295 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:49.564031 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:49.564037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:49.564043 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:49.564194 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:49.564610 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:49.564613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:49.564615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:49.565094 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:49.565215 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:49.567931 (+ 2716us) server_negotiation.cc:299] Negotiation successful
0925 22:57:49.568273 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2382,"thread_start_us":121,"threads_started":1}
I20260925 22:57:49.569119 4503 raft_consensus.cc:348] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } }
I20260925 22:57:49.569730 4503 raft_consensus.cc:729] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622a71eba47c4aaeac3c04f2c3eb199c, State: Initialized, Role: FOLLOWER
I20260925 22:57:49.570294 4503 consensus_queue.cc:260] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } }
I20260925 22:57:49.571420 4503 ts_tablet_manager.cc:1428] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:57:49.571789 4611 heartbeater.cc:347] Connected to a master server at 127.20.114.254:37499
I20260925 22:57:49.572001 4611 heartbeater.cc:464] Registering TS with master...
I20260925 22:57:49.572472 4611 heartbeater.cc:511] Master 127.20.114.254:37499 requested a full tablet report, sending...
I20260925 22:57:49.575474 4423 ts_manager.cc:194] Registered new tserver with Master: 622a71eba47c4aaeac3c04f2c3eb199c (127.20.114.193:36535)
I20260925 22:57:49.579607 4423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:58041
I20260925 22:57:49.582584 4611 heartbeater.cc:503] Master 127.20.114.254:37499 was elected leader, sending a full tablet report...
Sep 25 22:57:49 dist-test-slave-x1lh krb5kdc[3844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377069, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for kudu/127.20.114.193@KRBTEST.COM
I20260925 22:57:49.590714 4619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:49.579232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50610 (local address 127.20.114.193:36535)
0925 22:57:49.579696 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:49.579707 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:49.580111 (+ 404us) server_negotiation.cc:407] Connection header received
0925 22:57:49.580359 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:49.580366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:49.580587 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:49.580716 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:49.581528 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:49.582328 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:49.583457 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:49.583813 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:49.583943 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:49.586468 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:49.586485 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:49.586496 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:49.586518 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:49.588334 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:49.589182 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:49.589188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:49.589190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:49.589281 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:49.589576 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:49.589579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:49.589581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:49.589858 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:49.590011 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:49.590378 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 22:57:49.590555 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":126,"threads_started":1}
I20260925 22:57:49.590714 4618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:49.579145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:36535 (local address 127.0.0.1:50610)
0925 22:57:49.579955 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:49.579993 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:49.580187 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:49.580916 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:49.580923 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:49.581281 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:49.581397 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:49.581404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:49.582472 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:49.582476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:49.583325 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:49.583338 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:49.583945 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:49.583949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:49.584064 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:49.584720 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:49.584732 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:49.586323 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:49.588609 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:49.588622 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:49.588627 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:49.589063 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:49.589385 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:49.589388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:49.589390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:49.589478 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:49.589983 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:49.589989 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:49.590148 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 22:57:49.590283 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 22:57:49.590518 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":111,"threads_started":1}
I20260925 22:57:49.592167 4545 tablet_service.cc:1514] Processing DeleteTablet for tablet f15bc7aaea87443db116746273ff1310 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:57:29 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:50610
I20260925 22:57:49.593106 4620 tablet_replica.cc:330] stopping tablet replica
I20260925 22:57:49.593443 4620 raft_consensus.cc:2229] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:57:49.593706 4620 raft_consensus.cc:2258] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:57:49.594741 4620 ts_tablet_manager.cc:1905] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:57:49.598399 4620 ts_tablet_manager.cc:1918] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:57:49.598485 4620 log.cc:1196] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/wal/wals/f15bc7aaea87443db116746273ff1310
I20260925 22:57:49.598742 4620 ts_tablet_manager.cc:1939] T f15bc7aaea87443db116746273ff1310 P 622a71eba47c4aaeac3c04f2c3eb199c: Deleting consensus metadata
I20260925 22:57:49.600589 4410 catalog_manager.cc:4841] TS 622a71eba47c4aaeac3c04f2c3eb199c (127.20.114.193:36535): tablet f15bc7aaea87443db116746273ff1310 (table test-table [id=aef98bd7989841fca1d06fbc552e109f]) successfully deleted
W20260925 22:57:49.672609 4615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:49.672963 4615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:49.673075 4615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:49.677305 4615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:49.677395 4615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:49.677490 4615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:57:49.682958 4615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:44883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=34483
--tserver_master_addrs=127.20.114.254:37499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:49.684125 4615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:49.685251 4615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:57:49.695797 4628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:49.695892 4626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:49.695816 4625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:49.696184 4615 server_base.cc:1034] running on GCE node
I20260925 22:57:49.696595 4615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:49.697381 4615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:49.698585 4615 hybrid_clock.cc:648] HybridClock initialized: now 1790377069698563 us; error 38 us; skew 500 ppm
Sep 25 22:57:49 dist-test-slave-x1lh krb5kdc[3844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377069, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:49.701535 4615 init.cc:377] Logged in from keytab as kudu/127.20.114.194@KRBTEST.COM (short username kudu)
I20260925 22:57:49.702517 4615 webserver.cc:460] Webserver started at http://127.20.114.194:34483/ using document root <none> and password file <none>
I20260925 22:57:49.703127 4615 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:49.703205 4615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:49.707155 4615 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:57:49.709337 4635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:49.710623 4615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:57:49.710719 4615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "7e931ecace514df79840014f6ecb2038"
format_stamp: "Formatted at 2026-09-25 22:57:28 on dist-test-slave-x1lh"
server_key: "adbb15d9f9127ed576ea325d9f4e419b"
server_key_iv: "4a96185726dbecaa2aa490ebc6a3c06e"
server_key_version: "encryptionkey@0"
I20260925 22:57:49.711251 4615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:49.730374 4615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:49.733580 4615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:49.733736 4615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:49.734275 4615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:57:49.735064 4615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:57:49.735116 4615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:49.735157 4615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:57:49.735175 4615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:49.745100 4615 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:44883
I20260925 22:57:49.745123 4748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:44883 every 8 connection(s)
I20260925 22:57:49.746189 4615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:57:49.753666 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 4615
I20260925 22:57:49.754184 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:33565
--local_ip_for_outbound_sockets=127.20.114.195
--tserver_master_addrs=127.20.114.254:37499
--webserver_port=41493
--webserver_interface=127.20.114.195
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:57:49 dist-test-slave-x1lh krb5kdc[3844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377069, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:57:49.761729 4613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:49.748010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:58565 (local address 127.20.114.254:37499)
0925 22:57:49.748157 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:49.748161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:49.749206 (+ 1045us) server_negotiation.cc:407] Connection header received
0925 22:57:49.750690 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:49.750694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:49.750757 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:49.750873 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:49.752118 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:49.752657 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:49.753884 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:49.754199 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:49.754318 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:49.757268 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:49.757276 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:49.757282 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:49.757317 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:49.759092 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:49.759644 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:49.759650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:49.759655 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:49.759759 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:49.760229 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:49.760236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:49.760240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:49.760531 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:49.760647 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:49.761292 (+ 645us) server_negotiation.cc:299] Negotiation successful
0925 22:57:49.761466 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:57:49.762459 4751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:49.748463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.194:58565)
0925 22:57:49.749047 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:49.749092 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:49.750462 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:49.751078 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:49.751085 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:49.751468 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:49.751958 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:49.751973 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:49.752792 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:49.752795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:49.753741 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:49.753756 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:49.754379 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:49.754384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:49.754556 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:49.755272 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:49.755321 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:49.757078 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:49.759214 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:49.759219 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:49.759234 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:49.759523 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:49.759947 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:49.759949 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:49.759951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:49.760114 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:49.760651 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:49.760656 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:49.761021 (+ 365us) client_negotiation.cc:769] Sending connection context
0925 22:57:49.761187 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 22:57:49.761466 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":136,"threads_started":1}
I20260925 22:57:49.763747 4749 heartbeater.cc:347] Connected to a master server at 127.20.114.254:37499
I20260925 22:57:49.764031 4749 heartbeater.cc:464] Registering TS with master...
I20260925 22:57:49.764542 4749 heartbeater.cc:511] Master 127.20.114.254:37499 requested a full tablet report, sending...
I20260925 22:57:49.765656 4423 ts_manager.cc:194] Registered new tserver with Master: 7e931ecace514df79840014f6ecb2038 (127.20.114.194:44883)
I20260925 22:57:49.766281 4423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:58565
W20260925 22:57:49.873611 4752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:57:49.873906 4752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:57:49.873952 4752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:57:49.878378 4752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:57:49.878466 4752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:57:49.878561 4752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:57:49.884485 4752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:33565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=41493
--tserver_master_addrs=127.20.114.254:37499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:57:49.885757 4752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:57:49.886858 4752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:57:49.897512 4760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:49.897512 4757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:57:49.897567 4758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:57:49.897696 4752 server_base.cc:1034] running on GCE node
I20260925 22:57:49.898268 4752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:57:49.898967 4752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:57:49.900139 4752 hybrid_clock.cc:648] HybridClock initialized: now 1790377069900121 us; error 29 us; skew 500 ppm
Sep 25 22:57:49 dist-test-slave-x1lh krb5kdc[3844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377069, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:57:49.903182 4752 init.cc:377] Logged in from keytab as kudu/127.20.114.195@KRBTEST.COM (short username kudu)
I20260925 22:57:49.904125 4752 webserver.cc:460] Webserver started at http://127.20.114.195:41493/ using document root <none> and password file <none>
I20260925 22:57:49.904698 4752 fs_manager.cc:359] Metadata directory not provided
I20260925 22:57:49.904771 4752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:57:49.908829 4752 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260925 22:57:49.911108 4767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:49.912612 4752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:57:49.912720 4752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "547ccf22dc044da08b3b8636fd64a2ab"
format_stamp: "Formatted at 2026-09-25 22:57:28 on dist-test-slave-x1lh"
server_key: "f9e23b273bbc3a797015ec0c12d6e599"
server_key_iv: "086b5481dce447e4fce9d144016016a2"
server_key_version: "encryptionkey@0"
I20260925 22:57:49.913339 4752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:57:49.948654 4752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:57:49.951958 4752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:57:49.952220 4752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:57:49.952939 4752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:57:49.953941 4752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:57:49.953994 4752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:49.954037 4752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:57:49.954057 4752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:57:49.964072 4752 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:33565
I20260925 22:57:49.964138 4880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:33565 every 8 connection(s)
I20260925 22:57:49.965196 4752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:57:49.972729 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 4752
Sep 25 22:57:49 dist-test-slave-x1lh krb5kdc[3844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377069, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:57:49.981863 4613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:49.967310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:39249 (local address 127.20.114.254:37499)
0925 22:57:49.967469 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:49.967474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:49.968520 (+ 1046us) server_negotiation.cc:407] Connection header received
0925 22:57:49.970120 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:49.970123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:49.970178 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:49.970261 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:49.971651 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:49.972289 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:49.974029 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:49.974270 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:49.974415 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:49.977595 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:49.977599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:49.977602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:49.977638 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:49.979206 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:49.979931 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:49.979938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:49.979944 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:49.980045 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:49.980508 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:49.980514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:49.980518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:49.980904 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:49.981025 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:49.981519 (+ 494us) server_negotiation.cc:299] Negotiation successful
0925 22:57:49.981650 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":64}
I20260925 22:57:49.982765 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:49.967815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.195:39249)
0925 22:57:49.968346 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:49.968394 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:49.969854 (+ 1460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:49.970490 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:49.970497 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:49.970914 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:57:49.971458 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:49.971474 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:49.972495 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:49.972499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:49.973873 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:49.973888 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:49.974447 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:49.974452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:49.974649 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:49.975353 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:57:49.975374 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:57:49.977404 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:57:49.979342 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:49.979348 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:49.979369 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:49.979738 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:49.980148 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:57:49.980151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:57:49.980152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:57:49.980360 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:57:49.981014 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:57:49.981019 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:57:49.981281 (+ 262us) client_negotiation.cc:769] Sending connection context
0925 22:57:49.981453 (+ 172us) client_negotiation.cc:240] Negotiation successful
0925 22:57:49.981781 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":110,"threads_started":1}
I20260925 22:57:49.983489 4885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:49.973626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36238 (local address 127.20.114.254:37499)
0925 22:57:49.973935 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:49.973941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:49.974075 (+ 134us) server_negotiation.cc:407] Connection header received
0925 22:57:49.974219 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:49.974222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:49.974275 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:49.974379 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:49.975247 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:49.976079 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:49.977372 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:49.977621 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:49.977738 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:49.979355 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:49.979359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:49.979361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:49.979385 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:49.981238 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:49.981769 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:49.981773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:49.981775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:49.981862 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:49.982523 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:49.982526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:49.982528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:49.982737 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:49.982964 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:49.983214 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 22:57:49.983314 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":230,"thread_start_us":99,"threads_started":1}
I20260925 22:57:49.984417 4881 heartbeater.cc:347] Connected to a master server at 127.20.114.254:37499
I20260925 22:57:49.984704 4881 heartbeater.cc:464] Registering TS with master...
I20260925 22:57:49.985313 4881 heartbeater.cc:511] Master 127.20.114.254:37499 requested a full tablet report, sending...
I20260925 22:57:49.986377 4423 ts_manager.cc:194] Registered new tserver with Master: 547ccf22dc044da08b3b8636fd64a2ab (127.20.114.195:33565)
I20260925 22:57:49.986985 4423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:39249
I20260925 22:57:49.997730 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:57:50.002434 4885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:49.998433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36244 (local address 127.20.114.254:37499)
0925 22:57:49.998601 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:49.998605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:49.998756 (+ 151us) server_negotiation.cc:407] Connection header received
0925 22:57:49.999033 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:49.999039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:49.999154 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:49.999281 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:57:49.999547 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:50.000084 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:50.001020 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:50.001363 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:50.001482 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:50.001826 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:57:50.001978 (+ 152us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 22:57:50.002289 (+ 311us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36244: token expired
Metrics: {"server-negotiator.queue_time_us":80}
W20260925 22:57:50.002537 4885 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36244: token expired
W20260925 22:57:50.002856 4353 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.20.114.254:37499)
I20260925 22:57:50.010857 4885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:50.003304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36254 (local address 127.20.114.254:37499)
0925 22:57:50.003500 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:50.003505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:50.003518 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:57:50.003638 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:50.003641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:50.003689 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:50.003770 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:57:50.004411 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:50.004984 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:50.005808 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:50.006036 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:50.006143 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:50.007522 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:57:50.007526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:57:50.007529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:57:50.007564 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:57:50.009127 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:50.009586 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:50.009590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:50.009592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:50.009676 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:57:50.009984 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:57:50.009987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:57:50.009988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:57:50.010175 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:57:50.010314 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:50.010556 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 22:57:50.010683 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260925 22:57:50.013777 4423 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:57:50.017413 4423 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:57:50.026178 4545 tablet_service.cc:1467] Processing CreateTablet for tablet 9d6019655bf944df947f36024708ab51 (DEFAULT_TABLE table=test-table [id=cc37377576644db0b4d8f97293fb645e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:57:50.026885 4545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d6019655bf944df947f36024708ab51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:57:50.029587 4503 tablet_bootstrap.cc:492] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c: Bootstrap starting.
I20260925 22:57:50.030409 4503 tablet_bootstrap.cc:654] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c: Neither blocks nor log segments found. Creating new log.
I20260925 22:57:50.031388 4618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:50.024984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:44883 (local address 127.0.0.1:40762)
0925 22:57:50.025137 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:50.025154 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:50.025241 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:50.026452 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:50.026455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:50.026471 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:57:50.026621 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:50.026636 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:50.028300 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:50.028304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:50.029645 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:50.029656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:50.030842 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:50.030845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:50.030969 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:50.031023 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 22:57:50.031113 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 22:57:50.031223 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":25}
I20260925 22:57:50.031466 4892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:50.025470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40762 (local address 127.20.114.194:44883)
0925 22:57:50.025909 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:50.025924 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:50.025949 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:57:50.026030 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:50.026036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:50.026266 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:50.026935 (+ 669us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:57:50.027108 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:50.028175 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:50.029820 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:50.030699 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:50.030830 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:50.030971 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:50.031167 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 22:57:50.031283 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":136,"threads_started":1}
I20260925 22:57:50.032115 4890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:50.025234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:33565 (local address 127.0.0.1:42172)
0925 22:57:50.025484 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:50.025498 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:50.025590 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:50.026542 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:50.026547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:50.026582 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:57:50.026680 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:50.026685 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:50.028186 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:50.028191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:50.029709 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:50.029719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:50.031663 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:50.031668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:50.031796 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:50.031833 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 22:57:50.031910 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 22:57:50.031998 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":68,"thread_start_us":122,"threads_started":1}
I20260925 22:57:50.032439 4503 tablet_bootstrap.cc:492] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c: No bootstrap required, opened a new log
I20260925 22:57:50.032542 4503 ts_tablet_manager.cc:1397] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260925 22:57:50.032668 4891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:50.025416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42172 (local address 127.20.114.195:33565)
0925 22:57:50.025910 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:50.025924 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:50.025948 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:57:50.026030 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:50.026036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:50.026266 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:50.026441 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:57:50.026819 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:50.028054 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:50.029853 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:50.030699 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:50.030830 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:50.030971 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:50.032380 (+ 1409us) server_negotiation.cc:299] Negotiation successful
0925 22:57:50.032476 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":173,"threads_started":1}
I20260925 22:57:50.033061 4503 raft_consensus.cc:348] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } }
I20260925 22:57:50.033192 4503 raft_consensus.cc:374] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:57:50.033231 4503 raft_consensus.cc:729] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622a71eba47c4aaeac3c04f2c3eb199c, State: Initialized, Role: FOLLOWER
I20260925 22:57:50.033358 4503 consensus_queue.cc:260] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } }
I20260925 22:57:50.033618 4503 ts_tablet_manager.cc:1428] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 22:57:50.034907 4683 tablet_service.cc:1467] Processing CreateTablet for tablet 9d6019655bf944df947f36024708ab51 (DEFAULT_TABLE table=test-table [id=cc37377576644db0b4d8f97293fb645e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:57:50.035444 4815 tablet_service.cc:1467] Processing CreateTablet for tablet 9d6019655bf944df947f36024708ab51 (DEFAULT_TABLE table=test-table [id=cc37377576644db0b4d8f97293fb645e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:57:50.036139 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d6019655bf944df947f36024708ab51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:57:50.036468 4815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d6019655bf944df947f36024708ab51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:57:50.044480 4894 tablet_bootstrap.cc:492] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab: Bootstrap starting.
I20260925 22:57:50.047235 4895 tablet_bootstrap.cc:492] T 9d6019655bf944df947f36024708ab51 P 7e931ecace514df79840014f6ecb2038: Bootstrap starting.
I20260925 22:57:50.047389 4894 tablet_bootstrap.cc:654] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab: Neither blocks nor log segments found. Creating new log.
I20260925 22:57:50.048851 4894 log.cc:824] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab: Log is configured to *not* fsync() on all Append() calls
I20260925 22:57:50.050333 4895 tablet_bootstrap.cc:654] T 9d6019655bf944df947f36024708ab51 P 7e931ecace514df79840014f6ecb2038: Neither blocks nor log segments found. Creating new log.
I20260925 22:57:50.051317 4895 log.cc:824] T 9d6019655bf944df947f36024708ab51 P 7e931ecace514df79840014f6ecb2038: Log is configured to *not* fsync() on all Append() calls
W20260925 22:57:50.052606 4612 tablet.cc:2369] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:57:50.052806 4894 tablet_bootstrap.cc:492] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab: No bootstrap required, opened a new log
I20260925 22:57:50.053054 4894 ts_tablet_manager.cc:1397] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 22:57:50.054121 4895 tablet_bootstrap.cc:492] T 9d6019655bf944df947f36024708ab51 P 7e931ecace514df79840014f6ecb2038: No bootstrap required, opened a new log
I20260925 22:57:50.054349 4895 ts_tablet_manager.cc:1397] T 9d6019655bf944df947f36024708ab51 P 7e931ecace514df79840014f6ecb2038: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:57:50.056500 4894 raft_consensus.cc:348] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } }
I20260925 22:57:50.056772 4894 raft_consensus.cc:374] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:57:50.056833 4894 raft_consensus.cc:729] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547ccf22dc044da08b3b8636fd64a2ab, State: Initialized, Role: FOLLOWER
I20260925 22:57:50.057336 4894 consensus_queue.cc:260] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } }
I20260925 22:57:50.057605 4895 raft_consensus.cc:348] T 9d6019655bf944df947f36024708ab51 P 7e931ecace514df79840014f6ecb2038 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } }
I20260925 22:57:50.058055 4881 heartbeater.cc:503] Master 127.20.114.254:37499 was elected leader, sending a full tablet report...
I20260925 22:57:50.058342 4894 ts_tablet_manager.cc:1428] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:57:50.057894 4895 raft_consensus.cc:374] T 9d6019655bf944df947f36024708ab51 P 7e931ecace514df79840014f6ecb2038 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:57:50.059163 4895 raft_consensus.cc:729] T 9d6019655bf944df947f36024708ab51 P 7e931ecace514df79840014f6ecb2038 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e931ecace514df79840014f6ecb2038, State: Initialized, Role: FOLLOWER
I20260925 22:57:50.059620 4895 consensus_queue.cc:260] T 9d6019655bf944df947f36024708ab51 P 7e931ecace514df79840014f6ecb2038 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } }
I20260925 22:57:50.060313 4749 heartbeater.cc:503] Master 127.20.114.254:37499 was elected leader, sending a full tablet report...
I20260925 22:57:50.060640 4895 ts_tablet_manager.cc:1428] T 9d6019655bf944df947f36024708ab51 P 7e931ecace514df79840014f6ecb2038: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260925 22:57:50.216434 4882 tablet.cc:2369] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:57:50.247470 4750 tablet.cc:2369] T 9d6019655bf944df947f36024708ab51 P 7e931ecace514df79840014f6ecb2038: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:57:50.299103 4898 raft_consensus.cc:482] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:57:50.299391 4898 raft_consensus.cc:504] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } }
I20260925 22:57:50.300740 4898 leader_election.cc:288] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e931ecace514df79840014f6ecb2038 (127.20.114.194:44883), 622a71eba47c4aaeac3c04f2c3eb199c (127.20.114.193:36535)
I20260925 22:57:50.304939 4892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:50.301245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:48325 (local address 127.20.114.194:44883)
0925 22:57:50.301391 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:50.301396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:50.301634 (+ 238us) server_negotiation.cc:407] Connection header received
0925 22:57:50.301791 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:50.301794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:50.301852 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:50.301958 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:57:50.302250 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:50.302749 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:50.303952 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:50.304371 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:50.304500 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:50.304555 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:50.304721 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 22:57:50.304789 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 22:57:50.304965 4900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:50.301078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:44883 (local address 127.20.114.195:48325)
0925 22:57:50.301530 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:50.301542 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:50.301673 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:50.301968 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:50.301971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:50.301988 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:57:50.302132 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:50.302137 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:50.302875 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:50.302882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:50.303825 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:50.303842 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:50.304497 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:50.304501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:50.304600 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:50.304628 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 22:57:50.304705 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 22:57:50.304778 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":75,"thread_start_us":109,"threads_started":1}
I20260925 22:57:50.305148 4901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:50.301341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:57557 (local address 127.20.114.193:36535)
0925 22:57:50.301594 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:50.301597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:50.301613 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:57:50.301666 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:50.301668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:50.301720 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:50.301848 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:57:50.302176 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:50.302718 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:50.303958 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:50.304529 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:50.304668 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:50.304849 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:50.304912 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 22:57:50.304985 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":78,"threads_started":1}
I20260925 22:57:50.305466 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:50.301078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:36535 (local address 127.20.114.195:57557)
0925 22:57:50.301198 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:50.301223 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 22:57:50.301325 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:57:50.301857 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:57:50.301860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:57:50.301879 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:57:50.302043 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:50.302056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:50.302840 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:50.302847 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:50.303829 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:57:50.303838 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:57:50.304666 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:57:50.304670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:57:50.304768 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:50.304789 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:57:50.304851 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 22:57:50.304950 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":89}
I20260925 22:57:50.305800 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6019655bf944df947f36024708ab51" candidate_uuid: "547ccf22dc044da08b3b8636fd64a2ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e931ecace514df79840014f6ecb2038" is_pre_election: true
I20260925 22:57:50.306129 4703 raft_consensus.cc:2454] T 9d6019655bf944df947f36024708ab51 P 7e931ecace514df79840014f6ecb2038 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 547ccf22dc044da08b3b8636fd64a2ab in term 0.
I20260925 22:57:50.306099 4565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6019655bf944df947f36024708ab51" candidate_uuid: "547ccf22dc044da08b3b8636fd64a2ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" is_pre_election: true
I20260925 22:57:50.306401 4565 raft_consensus.cc:2454] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 547ccf22dc044da08b3b8636fd64a2ab in term 0.
I20260925 22:57:50.306648 4768 leader_election.cc:302] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 547ccf22dc044da08b3b8636fd64a2ab, 7e931ecace514df79840014f6ecb2038; no voters:
I20260925 22:57:50.306957 4898 raft_consensus.cc:2781] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:57:50.307049 4898 raft_consensus.cc:482] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:57:50.307091 4898 raft_consensus.cc:3037] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:57:50.308156 4898 raft_consensus.cc:504] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } }
I20260925 22:57:50.308501 4898 leader_election.cc:288] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab [CANDIDATE]: Term 1 election: Requested vote from peers 7e931ecace514df79840014f6ecb2038 (127.20.114.194:44883), 622a71eba47c4aaeac3c04f2c3eb199c (127.20.114.193:36535)
I20260925 22:57:50.308818 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6019655bf944df947f36024708ab51" candidate_uuid: "547ccf22dc044da08b3b8636fd64a2ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e931ecace514df79840014f6ecb2038"
I20260925 22:57:50.308969 4703 raft_consensus.cc:3037] T 9d6019655bf944df947f36024708ab51 P 7e931ecace514df79840014f6ecb2038 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:57:50.308979 4565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6019655bf944df947f36024708ab51" candidate_uuid: "547ccf22dc044da08b3b8636fd64a2ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622a71eba47c4aaeac3c04f2c3eb199c"
I20260925 22:57:50.309082 4565 raft_consensus.cc:3037] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:57:50.310029 4565 raft_consensus.cc:2454] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 547ccf22dc044da08b3b8636fd64a2ab in term 1.
I20260925 22:57:50.310278 4703 raft_consensus.cc:2454] T 9d6019655bf944df947f36024708ab51 P 7e931ecace514df79840014f6ecb2038 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 547ccf22dc044da08b3b8636fd64a2ab in term 1.
I20260925 22:57:50.310352 4771 leader_election.cc:302] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 547ccf22dc044da08b3b8636fd64a2ab, 622a71eba47c4aaeac3c04f2c3eb199c; no voters:
I20260925 22:57:50.310524 4898 raft_consensus.cc:2781] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:57:50.310745 4898 raft_consensus.cc:686] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab [term 1 LEADER]: Becoming Leader. State: Replica: 547ccf22dc044da08b3b8636fd64a2ab, State: Running, Role: LEADER
I20260925 22:57:50.311123 4898 consensus_queue.cc:237] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } }
I20260925 22:57:50.314718 4422 catalog_manager.cc:5495] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 547ccf22dc044da08b3b8636fd64a2ab (127.20.114.195). New cstate: current_term: 1 leader_uuid: "547ccf22dc044da08b3b8636fd64a2ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } health_report { overall_health: UNKNOWN } } }
I20260925 22:57:50.329119 4891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:50.325662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42174 (local address 127.20.114.195:33565)
0925 22:57:50.325791 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:57:50.325796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:57:50.325810 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:57:50.325893 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:57:50.325895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:57:50.325946 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:57:50.326061 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:57:50.326341 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:50.326917 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:50.327738 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:57:50.327950 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:57:50.328041 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:57:50.328282 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:57:50.328698 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:57:50.328795 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:57:50.328852 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 22:57:50.328975 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:57:50.335412 4703 raft_consensus.cc:1261] T 9d6019655bf944df947f36024708ab51 P 7e931ecace514df79840014f6ecb2038 [term 1 FOLLOWER]: Refusing update from remote peer 547ccf22dc044da08b3b8636fd64a2ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:57:50.335419 4565 raft_consensus.cc:1261] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [term 1 FOLLOWER]: Refusing update from remote peer 547ccf22dc044da08b3b8636fd64a2ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:57:50.336143 4898 consensus_queue.cc:1035] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:57:50.336357 4902 consensus_queue.cc:1035] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:57:50.353291 4423 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:57:50.353538 4423 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:57:50.356410 4423 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115: Sending DeleteTablet for 3 replicas of tablet 9d6019655bf944df947f36024708ab51
I20260925 22:57:50.357276 4545 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d6019655bf944df947f36024708ab51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:57:50 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:50610
I20260925 22:57:50.357293 4815 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d6019655bf944df947f36024708ab51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:57:50 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:42172
I20260925 22:57:50.357882 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 4461
I20260925 22:57:50.358016 4913 tablet_replica.cc:330] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c: stopping tablet replica
I20260925 22:57:50.358192 4913 raft_consensus.cc:2229] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:57:50.357904 4683 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d6019655bf944df947f36024708ab51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:57:50 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:40762
I20260925 22:57:50.358999 4915 tablet_replica.cc:330] T 9d6019655bf944df947f36024708ab51 P 7e931ecace514df79840014f6ecb2038: stopping tablet replica
I20260925 22:57:50.359047 4914 tablet_replica.cc:330] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab: stopping tablet replica
I20260925 22:57:50.359421 4915 raft_consensus.cc:2229] T 9d6019655bf944df947f36024708ab51 P 7e931ecace514df79840014f6ecb2038 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:57:50.359444 4914 raft_consensus.cc:2229] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:57:50.359869 4915 raft_consensus.cc:2258] T 9d6019655bf944df947f36024708ab51 P 7e931ecace514df79840014f6ecb2038 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:57:50.359941 4914 raft_consensus.cc:2258] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:57:50.361780 4915 ts_tablet_manager.cc:1905] T 9d6019655bf944df947f36024708ab51 P 7e931ecace514df79840014f6ecb2038: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:57:50.361800 4914 ts_tablet_manager.cc:1905] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:57:50.364697 4915 ts_tablet_manager.cc:1918] T 9d6019655bf944df947f36024708ab51 P 7e931ecace514df79840014f6ecb2038: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:57:50.364902 4915 log.cc:1196] T 9d6019655bf944df947f36024708ab51 P 7e931ecace514df79840014f6ecb2038: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/9d6019655bf944df947f36024708ab51
I20260925 22:57:50.365401 4915 ts_tablet_manager.cc:1939] T 9d6019655bf944df947f36024708ab51 P 7e931ecace514df79840014f6ecb2038: Deleting consensus metadata
I20260925 22:57:50.365542 4914 ts_tablet_manager.cc:1918] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:57:50.365655 4914 log.cc:1196] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/9d6019655bf944df947f36024708ab51
I20260925 22:57:50.366158 4914 ts_tablet_manager.cc:1939] T 9d6019655bf944df947f36024708ab51 P 547ccf22dc044da08b3b8636fd64a2ab: Deleting consensus metadata
I20260925 22:57:50.367478 4407 catalog_manager.cc:4841] TS 547ccf22dc044da08b3b8636fd64a2ab (127.20.114.195:33565): tablet 9d6019655bf944df947f36024708ab51 (table test-table [id=cc37377576644db0b4d8f97293fb645e]) successfully deleted
I20260925 22:57:50.367707 4407 catalog_manager.cc:4841] TS 7e931ecace514df79840014f6ecb2038 (127.20.114.194:44883): tablet 9d6019655bf944df947f36024708ab51 (table test-table [id=cc37377576644db0b4d8f97293fb645e]) successfully deleted
W20260925 22:57:50.368630 4410 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.114.193:36535 (error 108)
I20260925 22:57:50.368775 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 4615
I20260925 22:57:50.370997 4890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:57:50.370658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:36535 (local address 127.0.0.1:50618)
0925 22:57:50.370813 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 22:57:50.370896 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.193:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 22:57:50.371304 4410 catalog_manager.cc:4584] TS 622a71eba47c4aaeac3c04f2c3eb199c (127.20.114.193:36535): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d6019655bf944df947f36024708ab51: Network error: Client connection negotiation failed: client connection to 127.20.114.193:36535: connect: Connection refused (error 111)
I20260925 22:57:50.375432 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 4752
I20260925 22:57:50.380779 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 4390
I20260925 22:58:10.387878 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:37499
--webserver_interface=127.20.114.254
--webserver_port=40287
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:37499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:10.510166 4917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:10.510535 4917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:10.510623 4917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:10.515477 4917 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 22:58:10.515568 4917 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 22:58:10.515594 4917 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 22:58:10.515617 4917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:58:10.515661 4917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:10.515705 4917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:10.515743 4917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:58:10.515776 4917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:58:10.521670 4917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:37499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:37499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=40287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:10.522863 4917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:10.523960 4917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:10.533864 4923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:10.533867 4922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:10.533914 4917 server_base.cc:1034] running on GCE node
W20260925 22:58:10.533867 4925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:10.535000 4917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:10.536309 4917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:10.537495 4917 hybrid_clock.cc:648] HybridClock initialized: now 1790377090537451 us; error 50 us; skew 500 ppm
Sep 25 22:58:10 dist-test-slave-x1lh krb5kdc[3844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377090, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:10.546453 4917 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:58:10.547520 4917 webserver.cc:460] Webserver started at http://127.20.114.254:40287/ using document root <none> and password file <none>
I20260925 22:58:10.548115 4917 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:10.548174 4917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:10.552472 4917 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:58:10.555068 4932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:10.556525 4917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 22:58:10.556666 4917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "0351ec2f6f864dcfa85800eb53556115"
format_stamp: "Formatted at 2026-09-25 22:57:28 on dist-test-slave-x1lh"
server_key: "19120fcff5ef12ed91e7b80d854f1a31"
server_key_iv: "1efbb08296b3845f62b31883fcb9f014"
server_key_version: "encryptionkey@0"
I20260925 22:58:10.557368 4917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:10.586285 4917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:10.590207 4917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:10.590449 4917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:10.599164 4917 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:37499
I20260925 22:58:10.599480 4984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:37499 every 8 connection(s)
I20260925 22:58:10.600499 4917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:58:10.605594 4985 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:58:10.606228 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 4917
I20260925 22:58:10.606751 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:36535
--local_ip_for_outbound_sockets=127.20.114.193
--tserver_master_addrs=127.20.114.254:37499
--webserver_port=44881
--webserver_interface=127.20.114.193
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:58:10.607952 4985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115: Bootstrap starting.
I20260925 22:58:10.628268 4985 log.cc:824] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:10.640316 4985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:58:10.640800 4985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115: Bootstrap complete.
I20260925 22:58:10.645197 4985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0351ec2f6f864dcfa85800eb53556115" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37499 } }
I20260925 22:58:10.645677 4985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0351ec2f6f864dcfa85800eb53556115, State: Initialized, Role: FOLLOWER
I20260925 22:58:10.646224 4985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0351ec2f6f864dcfa85800eb53556115" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37499 } }
I20260925 22:58:10.646385 4985 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:58:10.646487 4985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:58:10.646624 4985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 2 FOLLOWER]: Advancing to term 3
I20260925 22:58:10.648703 4985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0351ec2f6f864dcfa85800eb53556115" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37499 } }
I20260925 22:58:10.649118 4985 leader_election.cc:302] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0351ec2f6f864dcfa85800eb53556115; no voters:
I20260925 22:58:10.649492 4985 leader_election.cc:288] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 22:58:10.649595 4989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 22:58:10.649812 4989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [term 3 LEADER]: Becoming Leader. State: Replica: 0351ec2f6f864dcfa85800eb53556115, State: Running, Role: LEADER
I20260925 22:58:10.650188 4989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0351ec2f6f864dcfa85800eb53556115" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37499 } }
I20260925 22:58:10.650887 4985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:58:10.651749 4990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0351ec2f6f864dcfa85800eb53556115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0351ec2f6f864dcfa85800eb53556115" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37499 } } }
I20260925 22:58:10.651894 4990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:10.652295 4991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0351ec2f6f864dcfa85800eb53556115. Latest consensus state: current_term: 3 leader_uuid: "0351ec2f6f864dcfa85800eb53556115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0351ec2f6f864dcfa85800eb53556115" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 37499 } } }
I20260925 22:58:10.652446 4991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:10.652729 4998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:58:10.659222 4998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:58:10.660073 4998 catalog_manager.cc:1259] Loaded cluster ID: 2282e0f3236a45b5b943ed5041f35782
I20260925 22:58:10.660178 4998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:58:10.663300 4998 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:58:10.664165 4998 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115: Loaded TSK: 0
I20260925 22:58:10.664266 4998 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0351ec2f6f864dcfa85800eb53556115: Loaded TSK: 1
F20260925 22:58:10.665114 4998 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fad1f2bf9ad google::LogMessage::Fail() at ??:0
@ 0x7fad1f2c288a google::LogMessage::SendToLog() at ??:0
@ 0x7fad1f2bf4de google::LogMessage::Flush() at ??:0
@ 0x7fad1f2c0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fad21f8318d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fad21f7bf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fad29c247bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fad29bfe99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fad29bff178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fad29c40df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fad29c5f82b std::function<>::operator()() at ??:0
@ 0x7fad29bfeb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fad29bfff3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fad29bfc579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fad29c40640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fad215113d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fad21511c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fad215133f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fad21502512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fad22b466db start_thread at ??:0
@ 0x7fad1ddbd61f clone at ??:0
W20260925 22:58:10.760208 4987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:10.760591 4987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:10.760645 4987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:10.765535 4987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:10.765592 4987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:10.765658 4987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:58:10.771839 4987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:36535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=44881
--tserver_master_addrs=127.20.114.254:37499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:10.773036 4987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:10.774309 4987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:10.784941 5033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:10.784982 5032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:10.785220 5035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:10.785290 4987 server_base.cc:1034] running on GCE node
I20260925 22:58:10.785789 4987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:10.786558 4987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:10.787748 4987 hybrid_clock.cc:648] HybridClock initialized: now 1790377090787706 us; error 64 us; skew 500 ppm
Sep 25 22:58:10 dist-test-slave-x1lh krb5kdc[3844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377090, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:10.790975 4987 init.cc:377] Logged in from keytab as kudu/127.20.114.193@KRBTEST.COM (short username kudu)
I20260925 22:58:10.792100 4987 webserver.cc:460] Webserver started at http://127.20.114.193:44881/ using document root <none> and password file <none>
I20260925 22:58:10.792732 4987 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:10.792830 4987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:10.797417 4987 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 22:58:10.800010 5042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:10.801847 4987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:10.802029 4987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "622a71eba47c4aaeac3c04f2c3eb199c"
format_stamp: "Formatted at 2026-09-25 22:57:28 on dist-test-slave-x1lh"
server_key: "95cf380f1a587840b5ecb456b234d549"
server_key_iv: "7823b1059c7d2134640063d732c52d94"
server_key_version: "encryptionkey@0"
I20260925 22:58:10.802691 4987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:10.816047 4987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:10.819998 4987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:10.820379 4987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:10.821183 4987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:58:10.822531 5049 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:58:10.828699 4987 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:58:10.828944 4987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260925 22:58:10.829078 4987 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:58:10.831034 4987 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:58:10.831135 4987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 22:58:10.831489 5049 tablet_bootstrap.cc:492] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c: Bootstrap starting.
I20260925 22:58:10.843037 4987 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:36535
I20260925 22:58:10.844163 4987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:58:10.844887 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 4987
I20260925 22:58:10.845450 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:44883
--local_ip_for_outbound_sockets=127.20.114.194
--tserver_master_addrs=127.20.114.254:37499
--webserver_port=34483
--webserver_interface=127.20.114.194
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:58:10.845510 5156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:36535 every 8 connection(s)
I20260925 22:58:10.855511 5049 log.cc:824] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:10.865721 5049 tablet_bootstrap.cc:492] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:58:10.866276 5049 tablet_bootstrap.cc:492] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c: Bootstrap complete.
I20260925 22:58:10.867820 5049 ts_tablet_manager.cc:1397] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c: Time spent bootstrapping tablet: real 0.037s user 0.021s sys 0.012s
I20260925 22:58:10.871512 5049 raft_consensus.cc:348] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } }
I20260925 22:58:10.872135 5049 raft_consensus.cc:729] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622a71eba47c4aaeac3c04f2c3eb199c, State: Initialized, Role: FOLLOWER
I20260925 22:58:10.872757 5049 consensus_queue.cc:260] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } }
I20260925 22:58:10.874022 5049 ts_tablet_manager.cc:1428] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260925 22:58:10.930352 5160 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:58:10.847669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.193:46049)
0925 22:58:10.848244 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:10.848305 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:10.849541 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:10.929158 (+ 79617us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":193,"threads_started":1}
W20260925 22:58:10.931075 5157 heartbeater.cc:644] Failed to heartbeat to 127.20.114.254:37499 (0 consecutive failures): Network error: Failed to ping master at 127.20.114.254:37499: Client connection negotiation failed: client connection to 127.20.114.254:37499: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 22:58:10.932077 5160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:10.931666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.193:45027)
0925 22:58:10.931852 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:10.931986 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 22:58:10.933063 5160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:10.932725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.193:50833)
0925 22:58:10.932939 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:10.933003 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260925 22:58:10.933285 5157 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:58:10.969832 5159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:10.970142 5159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:10.970189 5159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:10.974710 5159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:10.974794 5159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:10.974861 5159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:58:10.980746 5159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:44883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=34483
--tserver_master_addrs=127.20.114.254:37499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:10.981985 5159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:10.983084 5159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:10.993818 5167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:10.993834 5170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:10.993836 5168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:10.994174 5159 server_base.cc:1034] running on GCE node
I20260925 22:58:10.994674 5159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:10.995421 5159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:10.996596 5159 hybrid_clock.cc:648] HybridClock initialized: now 1790377090996578 us; error 33 us; skew 500 ppm
Sep 25 22:58:10 dist-test-slave-x1lh krb5kdc[3844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377090, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:10.999701 5159 init.cc:377] Logged in from keytab as kudu/127.20.114.194@KRBTEST.COM (short username kudu)
I20260925 22:58:11.000684 5159 webserver.cc:460] Webserver started at http://127.20.114.194:34483/ using document root <none> and password file <none>
I20260925 22:58:11.001356 5159 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:11.001426 5159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:11.006907 5159 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 22:58:11.009692 5177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:11.011374 5159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:11.011551 5159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "7e931ecace514df79840014f6ecb2038"
format_stamp: "Formatted at 2026-09-25 22:57:28 on dist-test-slave-x1lh"
server_key: "adbb15d9f9127ed576ea325d9f4e419b"
server_key_iv: "4a96185726dbecaa2aa490ebc6a3c06e"
server_key_version: "encryptionkey@0"
I20260925 22:58:11.012204 5159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:11.027016 5159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:11.030606 5159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:11.030826 5159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:11.031504 5159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:58:11.032501 5159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:58:11.032553 5159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:11.032596 5159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:58:11.032636 5159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:11.042179 5159 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:44883
I20260925 22:58:11.042246 5290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:44883 every 8 connection(s)
I20260925 22:58:11.043347 5159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:58:11.045665 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 5159
I20260925 22:58:11.046180 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:33565
--local_ip_for_outbound_sockets=127.20.114.195
--tserver_master_addrs=127.20.114.254:37499
--webserver_port=41493
--webserver_interface=127.20.114.195
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:58:11.048553 5293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:11.046085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.194:52677)
0925 22:58:11.047182 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:11.047311 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":909,"thread_start_us":171,"threads_started":1}
W20260925 22:58:11.049202 5291 heartbeater.cc:644] Failed to heartbeat to 127.20.114.254:37499 (0 consecutive failures): Network error: Failed to ping master at 127.20.114.254:37499: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
I20260925 22:58:11.050056 5293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:11.049704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.194:40101)
0925 22:58:11.049888 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:11.049961 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 22:58:11.050859 5293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:11.050575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.194:58865)
0925 22:58:11.050702 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:11.050777 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 22:58:11.051071 5291 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:58:11.164536 5294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:11.164889 5294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:11.164940 5294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:11.169166 5294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:11.169255 5294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:11.169318 5294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:58:11.174997 5294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:33565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=41493
--tserver_master_addrs=127.20.114.254:37499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:11.176147 5294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:11.177196 5294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:11.188126 5299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:11.188129 5302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:11.188308 5300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:11.188918 5294 server_base.cc:1034] running on GCE node
I20260925 22:58:11.189427 5294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:11.190225 5294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:11.191443 5294 hybrid_clock.cc:648] HybridClock initialized: now 1790377091191413 us; error 44 us; skew 500 ppm
Sep 25 22:58:11 dist-test-slave-x1lh krb5kdc[3844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377091, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:11.195622 5294 init.cc:377] Logged in from keytab as kudu/127.20.114.195@KRBTEST.COM (short username kudu)
I20260925 22:58:11.196805 5294 webserver.cc:460] Webserver started at http://127.20.114.195:41493/ using document root <none> and password file <none>
I20260925 22:58:11.197588 5294 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:11.197705 5294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:11.202984 5294 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260925 22:58:11.205621 5309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:11.207388 5294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:11.207564 5294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "547ccf22dc044da08b3b8636fd64a2ab"
format_stamp: "Formatted at 2026-09-25 22:57:28 on dist-test-slave-x1lh"
server_key: "f9e23b273bbc3a797015ec0c12d6e599"
server_key_iv: "086b5481dce447e4fce9d144016016a2"
server_key_version: "encryptionkey@0"
I20260925 22:58:11.208223 5294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:11.223387 5294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:11.227268 5294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:11.227684 5294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:11.228489 5294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:58:11.229491 5294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:58:11.229573 5294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:11.229645 5294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:58:11.229688 5294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:11.241001 5294 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:33565
I20260925 22:58:11.241039 5422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:33565 every 8 connection(s)
I20260925 22:58:11.242203 5294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:58:11.244474 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 5294
I20260925 22:58:11.247198 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:11.244911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.195:58313)
0925 22:58:11.245607 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:11.245791 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"spinlock_wait_cycles":28416,"thread_start_us":331,"threads_started":1}
W20260925 22:58:11.247704 5423 heartbeater.cc:644] Failed to heartbeat to 127.20.114.254:37499 (0 consecutive failures): Network error: Failed to ping master at 127.20.114.254:37499: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
I20260925 22:58:11.248548 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:11.248248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.195:57325)
0925 22:58:11.248424 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:11.248485 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 22:58:11.249456 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:11.249177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.195:48719)
0925 22:58:11.249308 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:11.249394 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 22:58:11.249662 5423 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 22:58:11.934695 5427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:11.934074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.193:54111)
0925 22:58:11.934426 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:11.934504 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":79,"threads_started":1}
I20260925 22:58:12.052438 5428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:12.051909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.194:38261)
0925 22:58:12.052231 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:12.052308 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":83,"threads_started":1}
I20260925 22:58:12.250937 5429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:12.250423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.195:35947)
0925 22:58:12.250734 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:12.250804 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":80,"threads_started":1}
I20260925 22:58:12.290890 5430 raft_consensus.cc:482] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:58:12.291301 5430 raft_consensus.cc:504] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } }
I20260925 22:58:12.292531 5430 leader_election.cc:288] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7e931ecace514df79840014f6ecb2038 (127.20.114.194:44883), 547ccf22dc044da08b3b8636fd64a2ab (127.20.114.195:33565)
Sep 25 22:58:12 dist-test-slave-x1lh krb5kdc[3844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377090, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.195@KRBTEST.COM
Sep 25 22:58:12 dist-test-slave-x1lh krb5kdc[3844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377090, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.194@KRBTEST.COM
I20260925 22:58:12.308653 5433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:12.293494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:48813 (local address 127.20.114.195:33565)
0925 22:58:12.293805 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:12.293824 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:12.293855 (+ 31us) server_negotiation.cc:407] Connection header received
0925 22:58:12.294485 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:12.294491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:12.294780 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:12.295072 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:12.296345 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:12.297429 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:12.298744 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:12.299278 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:12.299473 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:12.303083 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:12.303089 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:12.303105 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:12.303144 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:12.305398 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:12.306263 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:12.306269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:12.306273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:12.306420 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:12.306924 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:12.306927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:12.306928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:12.307421 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:12.307656 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:12.308175 (+ 519us) server_negotiation.cc:299] Negotiation successful
0925 22:58:12.308455 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":97,"threads_started":1}
I20260925 22:58:12.308635 5432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:12.293208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:33565 (local address 127.20.114.193:48813)
0925 22:58:12.293526 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:12.293537 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:12.293638 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:12.295047 (+ 1409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:12.295053 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:12.295643 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:58:12.296141 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:12.296157 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:12.297609 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:12.297611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:12.298609 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:12.298620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:12.299427 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:12.299431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:12.299645 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:12.300443 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:58:12.300467 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:58:12.302751 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:58:12.305585 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:58:12.305593 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:58:12.305612 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:58:12.306104 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:58:12.306574 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:58:12.306576 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:58:12.306578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:58:12.306760 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:58:12.307597 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:58:12.307600 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:58:12.307823 (+ 223us) client_negotiation.cc:769] Sending connection context
0925 22:58:12.308037 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 22:58:12.308432 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":166,"spinlock_wait_cycles":30592,"thread_start_us":84,"threads_started":1}
I20260925 22:58:12.308677 5427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:12.292841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:44883 (local address 127.20.114.193:57763)
0925 22:58:12.293022 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:12.293039 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:12.293156 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:12.295046 (+ 1890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:12.295053 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:12.295651 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:58:12.296140 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:12.296157 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:12.297591 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:12.297595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:12.298610 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:12.298620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:12.299444 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:12.299448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:12.299640 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:12.300468 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:58:12.300489 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:58:12.302912 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:58:12.305584 (+ 2672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:58:12.305593 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:58:12.305611 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:58:12.306086 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:58:12.306546 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:58:12.306549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:58:12.306551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:58:12.306745 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:58:12.307553 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:58:12.307562 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:58:12.307824 (+ 262us) client_negotiation.cc:769] Sending connection context
0925 22:58:12.308036 (+ 212us) client_negotiation.cc:240] Negotiation successful
0925 22:58:12.308522 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":381,"spinlock_wait_cycles":19840}
I20260925 22:58:12.308677 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:12.293042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:57763 (local address 127.20.114.194:44883)
0925 22:58:12.293324 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:12.293361 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:12.293430 (+ 69us) server_negotiation.cc:407] Connection header received
0925 22:58:12.294273 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:12.294281 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:12.294645 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:12.294984 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:12.296352 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:12.297430 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:12.298758 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:12.299306 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:12.299494 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:12.303219 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:12.303225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:12.303238 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:12.303277 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:12.305398 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:12.306249 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:12.306256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:12.306261 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:12.306412 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:12.306877 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:12.306881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:12.306885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:12.307427 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:12.307582 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:12.308177 (+ 595us) server_negotiation.cc:299] Negotiation successful
0925 22:58:12.308481 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
W20260925 22:58:12.311496 5043 leader_election.cc:341] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e931ecace514df79840014f6ecb2038 (127.20.114.194:44883): Not found: Tablet not found: 9d6019655bf944df947f36024708ab51
W20260925 22:58:12.311779 5043 leader_election.cc:341] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 547ccf22dc044da08b3b8636fd64a2ab (127.20.114.195:33565): Not found: Tablet not found: 9d6019655bf944df947f36024708ab51
I20260925 22:58:12.311877 5043 leader_election.cc:302] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 622a71eba47c4aaeac3c04f2c3eb199c; no voters: 547ccf22dc044da08b3b8636fd64a2ab, 7e931ecace514df79840014f6ecb2038
I20260925 22:58:12.312144 5430 raft_consensus.cc:2726] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:58:12.936210 5434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:12.935695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.193:60883)
0925 22:58:12.936037 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:12.936111 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":130,"threads_started":1}
I20260925 22:58:13.053838 5435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:13.053409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.194:47959)
0925 22:58:13.053678 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:13.053741 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":76,"threads_started":1}
I20260925 22:58:13.252262 5436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:13.251818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.195:42581)
0925 22:58:13.252104 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:13.252162 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":90,"threads_started":1}
I20260925 22:58:13.937754 5437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:13.937257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.193:45747)
0925 22:58:13.937578 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:13.937651 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":89,"threads_started":1}
I20260925 22:58:14.055141 5438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:14.054637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.194:58673)
0925 22:58:14.054975 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:14.055037 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":91,"threads_started":1}
I20260925 22:58:14.253532 5439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:14.253105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.195:36527)
0925 22:58:14.253371 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:14.253430 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":89,"threads_started":1}
I20260925 22:58:14.375730 5440 raft_consensus.cc:482] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:58:14.375910 5440 raft_consensus.cc:504] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } }
I20260925 22:58:14.376391 5440 leader_election.cc:288] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7e931ecace514df79840014f6ecb2038 (127.20.114.194:44883), 547ccf22dc044da08b3b8636fd64a2ab (127.20.114.195:33565)
W20260925 22:58:14.377215 5043 leader_election.cc:341] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e931ecace514df79840014f6ecb2038 (127.20.114.194:44883): Not found: Tablet not found: 9d6019655bf944df947f36024708ab51
W20260925 22:58:14.377331 5043 leader_election.cc:341] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 547ccf22dc044da08b3b8636fd64a2ab (127.20.114.195:33565): Not found: Tablet not found: 9d6019655bf944df947f36024708ab51
I20260925 22:58:14.377403 5043 leader_election.cc:302] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 622a71eba47c4aaeac3c04f2c3eb199c; no voters: 547ccf22dc044da08b3b8636fd64a2ab, 7e931ecace514df79840014f6ecb2038
I20260925 22:58:14.377525 5440 raft_consensus.cc:2726] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:58:14.939179 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:14.938704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.193:51231)
0925 22:58:14.939011 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:14.939078 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":88,"threads_started":1}
I20260925 22:58:15.056516 5442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:15.056064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.194:43391)
0925 22:58:15.056355 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:15.056419 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":101,"threads_started":1}
I20260925 22:58:15.254930 5443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:15.254428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.195:51905)
0925 22:58:15.254749 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:15.254816 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":95,"threads_started":1}
I20260925 22:58:15.941025 5444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:15.940348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.193:45647)
0925 22:58:15.940719 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:15.940850 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":98,"threads_started":1}
I20260925 22:58:16.058055 5445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:16.057498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.194:49433)
0925 22:58:16.057869 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:16.057948 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":99,"threads_started":1}
I20260925 22:58:16.256495 5446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:16.255979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.195:38653)
0925 22:58:16.256309 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:16.256388 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":161,"threads_started":1}
I20260925 22:58:16.764096 5447 raft_consensus.cc:482] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:58:16.764247 5447 raft_consensus.cc:504] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } }
I20260925 22:58:16.764603 5447 leader_election.cc:288] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7e931ecace514df79840014f6ecb2038 (127.20.114.194:44883), 547ccf22dc044da08b3b8636fd64a2ab (127.20.114.195:33565)
W20260925 22:58:16.765475 5043 leader_election.cc:341] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e931ecace514df79840014f6ecb2038 (127.20.114.194:44883): Not found: Tablet not found: 9d6019655bf944df947f36024708ab51
W20260925 22:58:16.765633 5043 leader_election.cc:341] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 547ccf22dc044da08b3b8636fd64a2ab (127.20.114.195:33565): Not found: Tablet not found: 9d6019655bf944df947f36024708ab51
I20260925 22:58:16.765702 5043 leader_election.cc:302] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 622a71eba47c4aaeac3c04f2c3eb199c; no voters: 547ccf22dc044da08b3b8636fd64a2ab, 7e931ecace514df79840014f6ecb2038
I20260925 22:58:16.765996 5447 raft_consensus.cc:2726] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:58:16.942714 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:16.942159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.193:35351)
0925 22:58:16.942498 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:16.942558 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":105,"threads_started":1}
I20260925 22:58:17.059464 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:17.058980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.194:41991)
0925 22:58:17.059281 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:17.059364 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":80,"threads_started":1}
I20260925 22:58:17.257970 5450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:17.257462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.195:33209)
0925 22:58:17.257756 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:17.257821 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":79,"threads_started":1}
I20260925 22:58:17.944116 5451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:17.943644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.193:55581)
0925 22:58:17.943950 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:17.944017 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":78,"threads_started":1}
I20260925 22:58:18.060971 5452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:18.060343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.194:46835)
0925 22:58:18.060671 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:18.060787 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":77,"threads_started":1}
I20260925 22:58:18.259533 5453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:18.258962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.195:42911)
0925 22:58:18.259342 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:18.259424 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":77,"threads_started":1}
I20260925 22:58:18.945538 5454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:18.945113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.193:54045)
0925 22:58:18.945373 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:18.945440 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":77,"threads_started":1}
I20260925 22:58:19.062389 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:19.061915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.194:39139)
0925 22:58:19.062218 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:19.062288 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":77,"threads_started":1}
I20260925 22:58:19.261173 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:19.260580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.195:36997)
0925 22:58:19.261001 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:19.261074 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":102,"threads_started":1}
I20260925 22:58:19.947113 5457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:19.946629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.193:37875)
0925 22:58:19.946947 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:19.947014 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":88,"threads_started":1}
I20260925 22:58:20.063722 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:20.063250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.194:45113)
0925 22:58:20.063554 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:20.063621 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":87,"threads_started":1}
I20260925 22:58:20.262501 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:20.262046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.195:48245)
0925 22:58:20.262336 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:20.262402 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":77,"threads_started":1}
I20260925 22:58:20.842201 5460 raft_consensus.cc:482] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:58:20.842368 5460 raft_consensus.cc:504] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e931ecace514df79840014f6ecb2038" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 44883 } } peers { permanent_uuid: "547ccf22dc044da08b3b8636fd64a2ab" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 33565 } } peers { permanent_uuid: "622a71eba47c4aaeac3c04f2c3eb199c" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36535 } }
I20260925 22:58:20.842941 5460 leader_election.cc:288] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7e931ecace514df79840014f6ecb2038 (127.20.114.194:44883), 547ccf22dc044da08b3b8636fd64a2ab (127.20.114.195:33565)
W20260925 22:58:20.843863 5043 leader_election.cc:341] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e931ecace514df79840014f6ecb2038 (127.20.114.194:44883): Not found: Tablet not found: 9d6019655bf944df947f36024708ab51
W20260925 22:58:20.844000 5043 leader_election.cc:341] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 547ccf22dc044da08b3b8636fd64a2ab (127.20.114.195:33565): Not found: Tablet not found: 9d6019655bf944df947f36024708ab51
I20260925 22:58:20.844084 5043 leader_election.cc:302] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 622a71eba47c4aaeac3c04f2c3eb199c; no voters: 547ccf22dc044da08b3b8636fd64a2ab, 7e931ecace514df79840014f6ecb2038
I20260925 22:58:20.844383 5460 raft_consensus.cc:2726] T 9d6019655bf944df947f36024708ab51 P 622a71eba47c4aaeac3c04f2c3eb199c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:58:20.948724 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:20.948200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.193:43925)
0925 22:58:20.948550 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:20.948621 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":90,"threads_started":1}
I20260925 22:58:21.065061 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:21.064596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.194:41903)
0925 22:58:21.064839 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:21.064964 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
I20260925 22:58:21.264338 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:21.263783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.195:58621)
0925 22:58:21.264170 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:21.264233 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":94,"threads_started":1}
I20260925 22:58:21.950204 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:21.949650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.193:56523)
0925 22:58:21.950018 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:21.950100 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":95,"threads_started":1}
I20260925 22:58:22.066310 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:22.065839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.194:41649)
0925 22:58:22.066139 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:22.066209 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":80,"threads_started":1}
I20260925 22:58:22.265875 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:22.265406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.195:35607)
0925 22:58:22.265702 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:22.265772 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":82,"threads_started":1}
I20260925 22:58:22.952199 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:22.951414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.193:50443)
0925 22:58:22.951855 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:22.952071 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":89,"threads_started":1}
I20260925 22:58:23.067575 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:23.067134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.194:40015)
0925 22:58:23.067413 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:23.067478 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":77,"threads_started":1}
I20260925 22:58:23.267241 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:23.266749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.195:60757)
0925 22:58:23.267057 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:23.267123 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":97,"threads_started":1}
I20260925 22:58:23.953722 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:23.953219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.193:59841)
0925 22:58:23.953530 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:23.953599 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":80,"threads_started":1}
I20260925 22:58:24.069104 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:24.068567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.194:51993)
0925 22:58:24.068849 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:24.068998 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":80,"threads_started":1}
I20260925 22:58:24.268699 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:24.268204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.195:44559)
0925 22:58:24.268530 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:24.268599 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":81,"threads_started":1}
I20260925 22:58:24.955250 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:24.954727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.193:56883)
0925 22:58:24.955028 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:24.955097 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":81,"threads_started":1}
I20260925 22:58:25.070488 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:25.070014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.194:43221)
0925 22:58:25.070303 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:25.070374 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":92,"threads_started":1}
I20260925 22:58:25.270246 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:25.269795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.195:55471)
0925 22:58:25.270086 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:25.270147 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":78,"threads_started":1}
I20260925 22:58:25.956923 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:25.956368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.193:40721)
0925 22:58:25.956661 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:25.956785 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":76,"threads_started":1}
I20260925 22:58:26.072234 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:26.071668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:37499 (local address 127.20.114.194:46301)
0925 22:58:26.072013 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:26.072074 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.254:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":92,"threads_started":1}
I20260925 22:58:26.252970 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 4987
I20260925 22:58:26.259285 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 5159
I20260925 22:58:26.265378 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 5294
2026-09-25T22:58:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59595 ms)
[----------] 30 tests from SecurityITest (250169 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:58:26 dist-test-slave-x1lh krb5kdc[5480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:58:26 dist-test-slave-x1lh krb5kdc[5480](info): set up 2 sockets
Sep 25 22:58:26 dist-test-slave-x1lh krb5kdc[5480](info): commencing operation
krb5kdc: starting...
W20260925 22:58:27.345443 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:58:27 dist-test-slave-x1lh krb5kdc[5480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:58:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:58:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:58:27.490821 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:34861
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:41099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:34861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:27.610623 5493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:27.611014 5493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:27.611109 5493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:27.615414 5493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:58:27.615543 5493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:27.615579 5493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:27.615618 5493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:58:27.615662 5493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:58:27.621490 5493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:41099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:34861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:34861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:27.622752 5493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:27.623922 5493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:58:27.633527 5493 server_base.cc:1034] running on GCE node
W20260925 22:58:27.633483 5499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:27.633515 5498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:27.633483 5501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:27.634593 5493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:27.635815 5493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:27.636948 5493 hybrid_clock.cc:648] HybridClock initialized: now 1790377107636912 us; error 50 us; skew 500 ppm
Sep 25 22:58:27 dist-test-slave-x1lh krb5kdc[5480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377107, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:27.640120 5493 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:58:27.641201 5493 webserver.cc:460] Webserver started at http://127.20.114.254:38599/ using document root <none> and password file <none>
I20260925 22:58:27.641834 5493 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:27.641930 5493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:27.642167 5493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:27.644361 5493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "9729d15cfec842d79a00b2aaa1184d63"
format_stamp: "Formatted at 2026-09-25 22:58:27 on dist-test-slave-x1lh"
server_key: "173ef0de8dc7a40701ac5f15d90932fb"
server_key_iv: "aa5f7d2696fc85f6b284897e639d855b"
server_key_version: "encryptionkey@0"
I20260925 22:58:27.645062 5493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "9729d15cfec842d79a00b2aaa1184d63"
format_stamp: "Formatted at 2026-09-25 22:58:27 on dist-test-slave-x1lh"
server_key: "173ef0de8dc7a40701ac5f15d90932fb"
server_key_iv: "aa5f7d2696fc85f6b284897e639d855b"
server_key_version: "encryptionkey@0"
I20260925 22:58:27.649348 5493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:58:27.652226 5508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:27.654047 5493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:27.654343 5493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "9729d15cfec842d79a00b2aaa1184d63"
format_stamp: "Formatted at 2026-09-25 22:58:27 on dist-test-slave-x1lh"
server_key: "173ef0de8dc7a40701ac5f15d90932fb"
server_key_iv: "aa5f7d2696fc85f6b284897e639d855b"
server_key_version: "encryptionkey@0"
I20260925 22:58:27.654502 5493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:27.677017 5493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:27.680586 5493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:27.680959 5493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:27.688561 5493 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:34861
I20260925 22:58:27.688577 5560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:34861 every 8 connection(s)
I20260925 22:58:27.689738 5493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:58:27.692775 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:27.698589 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 5493
I20260925 22:58:27.698628 5561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63: Bootstrap starting.
I20260925 22:58:27.698791 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:58:27.699075 20939 external_mini_cluster.cc:1428] Setting key 3d14daf4a7ed8e2d2b86753ff32318d1
I20260925 22:58:27.701522 5561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:27.702462 5561 log.cc:824] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:27.705029 5561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63: No bootstrap required, opened a new log
Sep 25 22:58:27 dist-test-slave-x1lh krb5kdc[5480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:58:27.709300 5561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9729d15cfec842d79a00b2aaa1184d63" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34861 } }
I20260925 22:58:27.709625 5561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:27.709695 5561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9729d15cfec842d79a00b2aaa1184d63, State: Initialized, Role: FOLLOWER
I20260925 22:58:27.710251 5561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9729d15cfec842d79a00b2aaa1184d63" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34861 } }
I20260925 22:58:27.710409 5561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:58:27.710476 5561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:58:27.710573 5561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:27.711550 5561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9729d15cfec842d79a00b2aaa1184d63" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34861 } }
I20260925 22:58:27.711925 5561 leader_election.cc:302] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9729d15cfec842d79a00b2aaa1184d63; no voters:
I20260925 22:58:27.712298 5561 leader_election.cc:288] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:58:27.712383 5566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:27.712575 5566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63 [term 1 LEADER]: Becoming Leader. State: Replica: 9729d15cfec842d79a00b2aaa1184d63, State: Running, Role: LEADER
I20260925 22:58:27.712967 5566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9729d15cfec842d79a00b2aaa1184d63" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34861 } }
I20260925 22:58:27.713719 5561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:58:27.714197 5566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9729d15cfec842d79a00b2aaa1184d63. Latest consensus state: current_term: 1 leader_uuid: "9729d15cfec842d79a00b2aaa1184d63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9729d15cfec842d79a00b2aaa1184d63" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34861 } } }
I20260925 22:58:27.714340 5566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:27.714893 5567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9729d15cfec842d79a00b2aaa1184d63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9729d15cfec842d79a00b2aaa1184d63" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 34861 } } }
I20260925 22:58:27.715149 5567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:27.716897 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:27.700645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56576 (local address 127.20.114.254:34861)
0925 22:58:27.701099 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:27.701119 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:27.701163 (+ 44us) server_negotiation.cc:407] Connection header received
0925 22:58:27.702102 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:27.702127 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:27.702584 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:27.702982 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:27.703945 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:27.705477 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:27.706393 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:27.707127 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:27.707304 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:27.710325 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:27.710331 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:27.710352 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:27.710387 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:27.712555 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:27.713206 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:27.713215 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:27.713226 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:27.713370 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:27.713673 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:27.713678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:27.713680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:27.714500 (+ 820us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:27.714725 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:27.715443 (+ 718us) server_negotiation.cc:299] Negotiation successful
0925 22:58:27.715804 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":164,"threads_started":1}
W20260925 22:58:27.718989 5581 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:58:27.719087 5581 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:58:27.719192 5570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:58:27.719990 5570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:58:27.725754 5570 catalog_manager.cc:1347] Generated new cluster ID: b207ff9282194575bf006eb93259d6e1
I20260925 22:58:27.725841 5570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:58:27.764532 5570 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:58:27.765715 5570 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:58:27.769968 5570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63: Generated new TSK 0
I20260925 22:58:27.770721 5570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.114.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
I20260925 22:58:27.810016 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:34861
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:41099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:27.927409 5587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:27.927701 5587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:27.927747 5587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:27.932015 5587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:27.932103 5587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:27.932168 5587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:58:27.938001 5587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:41099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:34861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:27.939180 5587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:27.940254 5587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:27.950555 5593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:27.950555 5595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:27.950589 5592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:27.950961 5587 server_base.cc:1034] running on GCE node
I20260925 22:58:27.951362 5587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:27.952041 5587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:27.953212 5587 hybrid_clock.cc:648] HybridClock initialized: now 1790377107953172 us; error 55 us; skew 500 ppm
Sep 25 22:58:27 dist-test-slave-x1lh krb5kdc[5480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377107, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:27.956092 5587 init.cc:377] Logged in from keytab as kudu/127.20.114.193@KRBTEST.COM (short username kudu)
I20260925 22:58:27.957041 5587 webserver.cc:460] Webserver started at http://127.20.114.193:45799/ using document root <none> and password file <none>
I20260925 22:58:27.957623 5587 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:27.957685 5587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:27.957858 5587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:27.960204 5587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "6616ff11012a4d99be6ac8b9c5b7e16a"
format_stamp: "Formatted at 2026-09-25 22:58:27 on dist-test-slave-x1lh"
server_key: "e9cb12e3ec855fa7c9b326f6de8ce9fa"
server_key_iv: "55c922ef28979ece578fa39a74d3f52b"
server_key_version: "encryptionkey@0"
I20260925 22:58:27.960752 5587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "6616ff11012a4d99be6ac8b9c5b7e16a"
format_stamp: "Formatted at 2026-09-25 22:58:27 on dist-test-slave-x1lh"
server_key: "e9cb12e3ec855fa7c9b326f6de8ce9fa"
server_key_iv: "55c922ef28979ece578fa39a74d3f52b"
server_key_version: "encryptionkey@0"
I20260925 22:58:27.964722 5587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260925 22:58:27.967381 5602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:27.968833 5587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:58:27.969038 5587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "6616ff11012a4d99be6ac8b9c5b7e16a"
format_stamp: "Formatted at 2026-09-25 22:58:27 on dist-test-slave-x1lh"
server_key: "e9cb12e3ec855fa7c9b326f6de8ce9fa"
server_key_iv: "55c922ef28979ece578fa39a74d3f52b"
server_key_version: "encryptionkey@0"
I20260925 22:58:27.969168 5587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:27.979367 5587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:27.982833 5587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:27.983052 5587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:27.983677 5587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:58:27.984594 5587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:58:27.984643 5587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:27.984685 5587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:58:27.984741 5587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:27.994689 5587 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:46075
I20260925 22:58:27.994720 5715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:46075 every 8 connection(s)
I20260925 22:58:27.995640 5587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:58:27.997202 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 5587
I20260925 22:58:27.997367 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:58:27.997723 20939 external_mini_cluster.cc:1428] Setting key c3e138c9c6af758de3990cdcf4a6c3d0
Sep 25 22:58:28 dist-test-slave-x1lh krb5kdc[5480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377107, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.114.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.194@KRBTEST.COM" created.
I20260925 22:58:28.013015 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:27.997635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:55903 (local address 127.20.114.254:34861)
0925 22:58:27.997811 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:27.997816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:27.998740 (+ 924us) server_negotiation.cc:407] Connection header received
0925 22:58:28.000400 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:28.000403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:28.000462 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:28.000538 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:28.002225 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.002881 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.004915 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.005179 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.005266 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:28.008858 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:28.008916 (+ 58us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:28.008922 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:28.008961 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:28.010403 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:28.011184 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:28.011188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:28.011190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:28.011290 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:28.011680 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:28.011687 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:28.011692 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:28.011964 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:28.012076 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:28.012644 (+ 568us) server_negotiation.cc:299] Negotiation successful
0925 22:58:28.012804 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 22:58:28.013913 5718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:27.998071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:34861 (local address 127.20.114.193:55903)
0925 22:58:27.998585 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:27.998645 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:28.000152 (+ 1507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:28.000724 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:28.000732 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:28.001243 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:58:28.001996 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:28.002021 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.003059 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.003065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:28.004699 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:28.004716 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.005367 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.005375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:28.005630 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:28.006411 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:58:28.006437 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:58:28.008636 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:58:28.010539 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:58:28.010547 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:58:28.010568 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:58:28.011048 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:58:28.011431 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:58:28.011433 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:58:28.011435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:58:28.011581 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:58:28.012097 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:58:28.012103 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:58:28.012375 (+ 272us) client_negotiation.cc:769] Sending connection context
0925 22:58:28.012597 (+ 222us) client_negotiation.cc:240] Negotiation successful
0925 22:58:28.012997 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":112,"threads_started":1}
I20260925 22:58:28.015306 5716 heartbeater.cc:347] Connected to a master server at 127.20.114.254:34861
I20260925 22:58:28.015625 5716 heartbeater.cc:464] Registering TS with master...
I20260925 22:58:28.016177 5716 heartbeater.cc:511] Master 127.20.114.254:34861 requested a full tablet report, sending...
I20260925 22:58:28.017922 5525 ts_manager.cc:194] Registered new tserver with Master: 6616ff11012a4d99be6ac8b9c5b7e16a (127.20.114.193:46075)
I20260925 22:58:28.019106 5525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:55903
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
I20260925 22:58:28.028532 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:34861
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:41099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:28.144616 5721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:28.145009 5721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:28.145112 5721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:28.149389 5721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:28.149459 5721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:28.149519 5721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:58:28.155254 5721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:41099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:34861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:28.156388 5721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:28.157491 5721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:28.167768 5729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:28.167771 5726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:28.168007 5721 server_base.cc:1034] running on GCE node
W20260925 22:58:28.167768 5727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:28.168625 5721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:28.169371 5721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:28.170557 5721 hybrid_clock.cc:648] HybridClock initialized: now 1790377108170532 us; error 38 us; skew 500 ppm
Sep 25 22:58:28 dist-test-slave-x1lh krb5kdc[5480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377108, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:28.173686 5721 init.cc:377] Logged in from keytab as kudu/127.20.114.194@KRBTEST.COM (short username kudu)
I20260925 22:58:28.174664 5721 webserver.cc:460] Webserver started at http://127.20.114.194:46313/ using document root <none> and password file <none>
I20260925 22:58:28.175256 5721 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:28.175348 5721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:28.175576 5721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:28.177891 5721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "6c82979c633a46a587cae59eb389a78e"
format_stamp: "Formatted at 2026-09-25 22:58:28 on dist-test-slave-x1lh"
server_key: "5796617b4ad1cd3e43f3de904ac40b2d"
server_key_iv: "60106a4481ce645e873b8feadd024c6d"
server_key_version: "encryptionkey@0"
I20260925 22:58:28.178493 5721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "6c82979c633a46a587cae59eb389a78e"
format_stamp: "Formatted at 2026-09-25 22:58:28 on dist-test-slave-x1lh"
server_key: "5796617b4ad1cd3e43f3de904ac40b2d"
server_key_iv: "60106a4481ce645e873b8feadd024c6d"
server_key_version: "encryptionkey@0"
I20260925 22:58:28.182533 5721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:58:28.185482 5736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:28.186877 5721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:28.187049 5721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "6c82979c633a46a587cae59eb389a78e"
format_stamp: "Formatted at 2026-09-25 22:58:28 on dist-test-slave-x1lh"
server_key: "5796617b4ad1cd3e43f3de904ac40b2d"
server_key_iv: "60106a4481ce645e873b8feadd024c6d"
server_key_version: "encryptionkey@0"
I20260925 22:58:28.187187 5721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:28.203146 5721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:28.206612 5721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:28.206846 5721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:28.207561 5721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:58:28.208801 5721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:58:28.208870 5721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:28.208976 5721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:58:28.209007 5721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:28.219096 5721 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:38721
I20260925 22:58:28.219117 5849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:38721 every 8 connection(s)
I20260925 22:58:28.220052 5721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:58:28.226095 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 5721
I20260925 22:58:28.226243 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:58:28.226531 20939 external_mini_cluster.cc:1428] Setting key 7dbc4b5160fbe71469d9f4ba60ee2107
Sep 25 22:58:28 dist-test-slave-x1lh krb5kdc[5480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377108, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:58:28.234836 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:28.221992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:59149 (local address 127.20.114.254:34861)
0925 22:58:28.222128 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:28.222133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:28.222984 (+ 851us) server_negotiation.cc:407] Connection header received
0925 22:58:28.224374 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:28.224377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:28.224437 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:28.224521 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:28.225789 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.226473 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.228070 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.228307 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.228426 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:28.231021 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:28.231024 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:28.231026 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:28.231051 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:28.232577 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:28.233218 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:28.233221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:28.233224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:28.233310 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:28.233681 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:28.233684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:28.233686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:28.233884 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:28.233993 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:28.234534 (+ 541us) server_negotiation.cc:299] Negotiation successful
0925 22:58:28.234670 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.20.114.195@KRBTEST.COM; defaulting to no policy
I20260925 22:58:28.235801 5852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:28.222395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:34861 (local address 127.20.114.194:59149)
0925 22:58:28.222833 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:28.222875 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:28.224156 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:28.224699 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:28.224706 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:28.225145 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:58:28.225641 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:28.225655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.226608 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.226612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:28.227918 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:28.227930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.228474 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.228478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:28.228658 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:28.229318 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:58:28.229335 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:58:28.230828 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:58:28.232716 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:58:28.232722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:58:28.232739 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:58:28.233111 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:58:28.233406 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:58:28.233412 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:58:28.233415 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:58:28.233575 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:58:28.233992 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:58:28.234001 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:58:28.234285 (+ 284us) client_negotiation.cc:769] Sending connection context
0925 22:58:28.234482 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 22:58:28.234842 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":110,"threads_started":1}
I20260925 22:58:28.237367 5850 heartbeater.cc:347] Connected to a master server at 127.20.114.254:34861
I20260925 22:58:28.237656 5850 heartbeater.cc:464] Registering TS with master...
I20260925 22:58:28.238320 5850 heartbeater.cc:511] Master 127.20.114.254:34861 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.195@KRBTEST.COM" created.
I20260925 22:58:28.239586 5525 ts_manager.cc:194] Registered new tserver with Master: 6c82979c633a46a587cae59eb389a78e (127.20.114.194:38721)
I20260925 22:58:28.240342 5525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:59149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
I20260925 22:58:28.257100 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:34861
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:41099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:28.374454 5855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:28.374738 5855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:28.374809 5855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:28.379108 5855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:28.379194 5855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:28.379287 5855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:58:28.385071 5855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:41099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:34861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:28.386245 5855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:28.387326 5855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:28.397861 5861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:28.397954 5863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:28.397990 5855 server_base.cc:1034] running on GCE node
W20260925 22:58:28.398212 5860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:28.398703 5855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:28.399500 5855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:28.400700 5855 hybrid_clock.cc:648] HybridClock initialized: now 1790377108400647 us; error 67 us; skew 500 ppm
Sep 25 22:58:28 dist-test-slave-x1lh krb5kdc[5480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377108, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:28.404003 5855 init.cc:377] Logged in from keytab as kudu/127.20.114.195@KRBTEST.COM (short username kudu)
I20260925 22:58:28.405119 5855 webserver.cc:460] Webserver started at http://127.20.114.195:38509/ using document root <none> and password file <none>
I20260925 22:58:28.405932 5855 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:28.406040 5855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:28.406291 5855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:28.408466 5855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "afe83aedf6ac4e5fb36c1ec70d821128"
format_stamp: "Formatted at 2026-09-25 22:58:28 on dist-test-slave-x1lh"
server_key: "6cfd7d3037608cf96e10d739fb39359e"
server_key_iv: "71221f09be9507916cadbab3575985bc"
server_key_version: "encryptionkey@0"
I20260925 22:58:28.409138 5855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "afe83aedf6ac4e5fb36c1ec70d821128"
format_stamp: "Formatted at 2026-09-25 22:58:28 on dist-test-slave-x1lh"
server_key: "6cfd7d3037608cf96e10d739fb39359e"
server_key_iv: "71221f09be9507916cadbab3575985bc"
server_key_version: "encryptionkey@0"
I20260925 22:58:28.413298 5855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:58:28.416399 5870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:28.417829 5855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:28.418047 5855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "afe83aedf6ac4e5fb36c1ec70d821128"
format_stamp: "Formatted at 2026-09-25 22:58:28 on dist-test-slave-x1lh"
server_key: "6cfd7d3037608cf96e10d739fb39359e"
server_key_iv: "71221f09be9507916cadbab3575985bc"
server_key_version: "encryptionkey@0"
I20260925 22:58:28.418196 5855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:28.444800 5855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:28.448525 5855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:28.448771 5855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:28.449515 5855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:58:28.450429 5855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:58:28.450505 5855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:28.450577 5855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:58:28.450627 5855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:28.460830 5855 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:37053
I20260925 22:58:28.460882 5983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:37053 every 8 connection(s)
I20260925 22:58:28.461843 5855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:58:28.465016 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 5855
I20260925 22:58:28.465170 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:58:28.465559 20939 external_mini_cluster.cc:1428] Setting key 46d7571a1d4aa6d3443afd13d1131fb4
Sep 25 22:58:28 dist-test-slave-x1lh krb5kdc[5480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377108, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:58:28.477854 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:28.463934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:53647 (local address 127.20.114.254:34861)
0925 22:58:28.464122 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:28.464126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:28.465240 (+ 1114us) server_negotiation.cc:407] Connection header received
0925 22:58:28.467196 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:28.467204 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:28.467265 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:28.467351 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:28.468678 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.469298 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.470530 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.470762 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.470879 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:28.473675 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:28.473679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:28.473681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:28.473711 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:28.475401 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:28.476052 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:28.476057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:28.476059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:28.476160 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:28.476562 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:28.476566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:28.476568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:28.476778 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:28.476920 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:28.477503 (+ 583us) server_negotiation.cc:299] Negotiation successful
0925 22:58:28.477670 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 22:58:28.478927 5986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:28.464385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:34861 (local address 127.20.114.195:53647)
0925 22:58:28.465034 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:28.465112 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:28.466967 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:28.467545 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:28.467553 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:28.467942 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:58:28.468528 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:28.468543 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.469437 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.469441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:28.470394 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:28.470404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.470930 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.470934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:28.471115 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:28.471719 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:58:28.471736 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:58:28.473367 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:58:28.475575 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:58:28.475590 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:58:28.475608 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:58:28.475930 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:58:28.476265 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:58:28.476268 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:58:28.476270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:58:28.476447 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:58:28.476926 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:58:28.476936 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:58:28.477233 (+ 297us) client_negotiation.cc:769] Sending connection context
0925 22:58:28.477448 (+ 215us) client_negotiation.cc:240] Negotiation successful
0925 22:58:28.477832 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":143,"threads_started":1}
I20260925 22:58:28.480161 5984 heartbeater.cc:347] Connected to a master server at 127.20.114.254:34861
I20260925 22:58:28.480443 5984 heartbeater.cc:464] Registering TS with master...
I20260925 22:58:28.481024 5984 heartbeater.cc:511] Master 127.20.114.254:34861 requested a full tablet report, sending...
I20260925 22:58:28.482095 5525 ts_manager.cc:194] Registered new tserver with Master: afe83aedf6ac4e5fb36c1ec70d821128 (127.20.114.195:37053)
I20260925 22:58:28.482704 5525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:53647
I20260925 22:58:28.493052 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:58:28.504247 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:28.495282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56586 (local address 127.20.114.254:34861)
0925 22:58:28.495490 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:28.495494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:28.495648 (+ 154us) server_negotiation.cc:407] Connection header received
0925 22:58:28.495875 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:28.495878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:28.495937 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:28.496029 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:28.496878 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.497429 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.498333 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.498562 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.498672 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:28.500559 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:28.500563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:28.500565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:28.500596 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:28.502177 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:28.502741 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:28.502745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:28.502747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:28.502848 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:28.503169 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:28.503174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:28.503176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:28.503465 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:28.503627 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:28.503919 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 22:58:28.504054 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260925 22:58:28.507993 5525 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:58:28.511251 5525 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:58:28.527266 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:28.520679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:37053 (local address 127.0.0.1:39764)
0925 22:58:28.521203 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:28.521254 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:28.521389 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:28.522654 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:28.522663 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:28.522692 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:28.522920 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:28.522939 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.524545 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.524548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:28.525838 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:28.525845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.526668 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.526671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:28.526788 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:28.526835 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 22:58:28.526948 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 22:58:28.527081 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":154,"spinlock_wait_cycles":1792,"thread_start_us":81,"threads_started":1}
I20260925 22:58:28.527436 5999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:28.521452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52672 (local address 127.20.114.194:38721)
0925 22:58:28.522170 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:28.522182 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:28.522305 (+ 123us) server_negotiation.cc:407] Connection header received
0925 22:58:28.522385 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:28.522392 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:28.522557 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:28.522883 (+ 326us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:28.523095 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.524388 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.525983 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.526751 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.526926 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:28.527041 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:28.527146 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 22:58:28.527245 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":111,"threads_started":1}
I20260925 22:58:28.527585 5996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:28.520848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39764 (local address 127.20.114.195:37053)
0925 22:58:28.521987 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:28.522004 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:28.522027 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:58:28.522104 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:28.522112 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:28.522485 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:28.522643 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:28.523066 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.524374 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.525954 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.526537 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.526681 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:28.526827 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:28.527331 (+ 504us) server_negotiation.cc:299] Negotiation successful
0925 22:58:28.527459 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1026,"thread_start_us":168,"threads_started":1}
I20260925 22:58:28.527264 5997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:28.520854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:38721 (local address 127.0.0.1:52672)
0925 22:58:28.521616 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:28.521632 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:28.521730 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:28.522717 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:28.522719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:28.522732 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:28.522996 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:28.523006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.524545 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.524548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:28.525841 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:28.525851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.526907 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.526914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:28.527030 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:28.527050 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:58:28.527114 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 22:58:28.527166 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":173,"thread_start_us":150,"threads_started":1}
I20260925 22:58:28.527925 5998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:28.521427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:46075 (local address 127.0.0.1:44838)
0925 22:58:28.521681 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:28.521693 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:28.521778 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:28.522895 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:28.522898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:28.522918 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:28.523032 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:28.523041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.524120 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.524127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:28.525569 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:28.525581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.526931 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.526935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:28.527073 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:28.527101 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 22:58:28.527173 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 22:58:28.527245 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":68,"thread_start_us":102,"threads_started":1}
I20260925 22:58:28.527973 6000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:28.521843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44838 (local address 127.20.114.193:46075)
0925 22:58:28.522228 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:28.522234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:28.522253 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:58:28.522316 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:28.522321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:28.522485 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:28.522643 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:28.523167 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.524003 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.525707 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.526596 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.526726 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:28.526872 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:28.527717 (+ 845us) server_negotiation.cc:299] Negotiation successful
0925 22:58:28.527831 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":135,"threads_started":1}
I20260925 22:58:28.530599 5918 tablet_service.cc:1467] Processing CreateTablet for tablet 426be197457646c496daabf4c177d35a (DEFAULT_TABLE table=test-table [id=375e42099779424d94fe45df9abece4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:58:28.530664 5650 tablet_service.cc:1467] Processing CreateTablet for tablet 426be197457646c496daabf4c177d35a (DEFAULT_TABLE table=test-table [id=375e42099779424d94fe45df9abece4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:58:28.530665 5784 tablet_service.cc:1467] Processing CreateTablet for tablet 426be197457646c496daabf4c177d35a (DEFAULT_TABLE table=test-table [id=375e42099779424d94fe45df9abece4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:58:28.531622 5918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 426be197457646c496daabf4c177d35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:28.531666 5784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 426be197457646c496daabf4c177d35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:28.531666 5650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 426be197457646c496daabf4c177d35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:28.539455 6002 tablet_bootstrap.cc:492] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e: Bootstrap starting.
I20260925 22:58:28.539455 6001 tablet_bootstrap.cc:492] T 426be197457646c496daabf4c177d35a P afe83aedf6ac4e5fb36c1ec70d821128: Bootstrap starting.
I20260925 22:58:28.541988 6002 tablet_bootstrap.cc:654] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:28.541988 6001 tablet_bootstrap.cc:654] T 426be197457646c496daabf4c177d35a P afe83aedf6ac4e5fb36c1ec70d821128: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:28.542631 6003 tablet_bootstrap.cc:492] T 426be197457646c496daabf4c177d35a P 6616ff11012a4d99be6ac8b9c5b7e16a: Bootstrap starting.
I20260925 22:58:28.543524 6001 log.cc:824] T 426be197457646c496daabf4c177d35a P afe83aedf6ac4e5fb36c1ec70d821128: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:28.543524 6002 log.cc:824] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:28.545994 6001 tablet_bootstrap.cc:492] T 426be197457646c496daabf4c177d35a P afe83aedf6ac4e5fb36c1ec70d821128: No bootstrap required, opened a new log
I20260925 22:58:28.546074 6003 tablet_bootstrap.cc:654] T 426be197457646c496daabf4c177d35a P 6616ff11012a4d99be6ac8b9c5b7e16a: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:28.546241 6001 ts_tablet_manager.cc:1397] T 426be197457646c496daabf4c177d35a P afe83aedf6ac4e5fb36c1ec70d821128: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:58:28.546571 6002 tablet_bootstrap.cc:492] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e: No bootstrap required, opened a new log
I20260925 22:58:28.546777 6002 ts_tablet_manager.cc:1397] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 22:58:28.547116 6003 log.cc:824] T 426be197457646c496daabf4c177d35a P 6616ff11012a4d99be6ac8b9c5b7e16a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:28.549491 6003 tablet_bootstrap.cc:492] T 426be197457646c496daabf4c177d35a P 6616ff11012a4d99be6ac8b9c5b7e16a: No bootstrap required, opened a new log
I20260925 22:58:28.549749 6003 ts_tablet_manager.cc:1397] T 426be197457646c496daabf4c177d35a P 6616ff11012a4d99be6ac8b9c5b7e16a: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:58:28.549780 6001 raft_consensus.cc:348] T 426be197457646c496daabf4c177d35a P afe83aedf6ac4e5fb36c1ec70d821128 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe83aedf6ac4e5fb36c1ec70d821128" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 37053 } } peers { permanent_uuid: "6c82979c633a46a587cae59eb389a78e" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 38721 } } peers { permanent_uuid: "6616ff11012a4d99be6ac8b9c5b7e16a" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 46075 } }
I20260925 22:58:28.550149 6001 raft_consensus.cc:374] T 426be197457646c496daabf4c177d35a P afe83aedf6ac4e5fb36c1ec70d821128 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:28.550226 6001 raft_consensus.cc:729] T 426be197457646c496daabf4c177d35a P afe83aedf6ac4e5fb36c1ec70d821128 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afe83aedf6ac4e5fb36c1ec70d821128, State: Initialized, Role: FOLLOWER
I20260925 22:58:28.550810 6001 consensus_queue.cc:260] T 426be197457646c496daabf4c177d35a P afe83aedf6ac4e5fb36c1ec70d821128 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe83aedf6ac4e5fb36c1ec70d821128" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 37053 } } peers { permanent_uuid: "6c82979c633a46a587cae59eb389a78e" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 38721 } } peers { permanent_uuid: "6616ff11012a4d99be6ac8b9c5b7e16a" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 46075 } }
I20260925 22:58:28.551115 6002 raft_consensus.cc:348] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe83aedf6ac4e5fb36c1ec70d821128" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 37053 } } peers { permanent_uuid: "6c82979c633a46a587cae59eb389a78e" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 38721 } } peers { permanent_uuid: "6616ff11012a4d99be6ac8b9c5b7e16a" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 46075 } }
I20260925 22:58:28.551476 6002 raft_consensus.cc:374] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:28.551543 6002 raft_consensus.cc:729] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c82979c633a46a587cae59eb389a78e, State: Initialized, Role: FOLLOWER
I20260925 22:58:28.552055 6001 ts_tablet_manager.cc:1428] T 426be197457646c496daabf4c177d35a P afe83aedf6ac4e5fb36c1ec70d821128: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:58:28.552126 6002 consensus_queue.cc:260] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe83aedf6ac4e5fb36c1ec70d821128" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 37053 } } peers { permanent_uuid: "6c82979c633a46a587cae59eb389a78e" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 38721 } } peers { permanent_uuid: "6616ff11012a4d99be6ac8b9c5b7e16a" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 46075 } }
I20260925 22:58:28.552446 5984 heartbeater.cc:503] Master 127.20.114.254:34861 was elected leader, sending a full tablet report...
I20260925 22:58:28.553020 5850 heartbeater.cc:503] Master 127.20.114.254:34861 was elected leader, sending a full tablet report...
I20260925 22:58:28.553460 6002 ts_tablet_manager.cc:1428] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 22:58:28.554701 6003 raft_consensus.cc:348] T 426be197457646c496daabf4c177d35a P 6616ff11012a4d99be6ac8b9c5b7e16a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe83aedf6ac4e5fb36c1ec70d821128" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 37053 } } peers { permanent_uuid: "6c82979c633a46a587cae59eb389a78e" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 38721 } } peers { permanent_uuid: "6616ff11012a4d99be6ac8b9c5b7e16a" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 46075 } }
I20260925 22:58:28.555001 6003 raft_consensus.cc:374] T 426be197457646c496daabf4c177d35a P 6616ff11012a4d99be6ac8b9c5b7e16a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:28.555070 6003 raft_consensus.cc:729] T 426be197457646c496daabf4c177d35a P 6616ff11012a4d99be6ac8b9c5b7e16a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6616ff11012a4d99be6ac8b9c5b7e16a, State: Initialized, Role: FOLLOWER
I20260925 22:58:28.555560 6003 consensus_queue.cc:260] T 426be197457646c496daabf4c177d35a P 6616ff11012a4d99be6ac8b9c5b7e16a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe83aedf6ac4e5fb36c1ec70d821128" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 37053 } } peers { permanent_uuid: "6c82979c633a46a587cae59eb389a78e" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 38721 } } peers { permanent_uuid: "6616ff11012a4d99be6ac8b9c5b7e16a" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 46075 } }
I20260925 22:58:28.556268 5716 heartbeater.cc:503] Master 127.20.114.254:34861 was elected leader, sending a full tablet report...
I20260925 22:58:28.556710 6003 ts_tablet_manager.cc:1428] T 426be197457646c496daabf4c177d35a P 6616ff11012a4d99be6ac8b9c5b7e16a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 22:58:28.601009 6008 raft_consensus.cc:482] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:58:28.601249 6008 raft_consensus.cc:504] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe83aedf6ac4e5fb36c1ec70d821128" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 37053 } } peers { permanent_uuid: "6c82979c633a46a587cae59eb389a78e" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 38721 } } peers { permanent_uuid: "6616ff11012a4d99be6ac8b9c5b7e16a" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 46075 } }
I20260925 22:58:28.602525 6008 leader_election.cc:288] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers afe83aedf6ac4e5fb36c1ec70d821128 (127.20.114.195:37053), 6616ff11012a4d99be6ac8b9c5b7e16a (127.20.114.193:46075)
I20260925 22:58:28.607851 6000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:28.602944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:52347 (local address 127.20.114.193:46075)
0925 22:58:28.603081 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:28.603086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:28.603300 (+ 214us) server_negotiation.cc:407] Connection header received
0925 22:58:28.603396 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:28.603402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:28.603480 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:28.603547 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:28.603819 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.604545 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.606396 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.607050 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.607194 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:28.607251 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:28.607622 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 22:58:28.607713 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:58:28.607888 5996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:28.602845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:38801 (local address 127.20.114.195:37053)
0925 22:58:28.602975 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:28.602982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:28.602999 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:58:28.603205 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:28.603208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:28.603255 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:28.603354 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:28.603643 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.604473 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.606396 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.607067 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.607194 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:28.607251 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:28.607622 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 22:58:28.607709 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:58:28.607890 5852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:28.602713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:37053 (local address 127.20.114.194:38801)
0925 22:58:28.602928 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:28.602943 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:28.603042 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:28.603381 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:28.603384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:28.603404 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:28.603507 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:28.603511 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.604623 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.604626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:28.606225 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:28.606238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.607251 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.607255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:28.607433 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:28.607487 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 22:58:28.607574 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 22:58:28.607694 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":252}
I20260925 22:58:28.607898 6010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:28.602802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:46075 (local address 127.20.114.194:52347)
0925 22:58:28.603212 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:28.603222 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:28.603299 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:28.603586 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:28.603591 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:28.603612 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:28.603721 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:28.603725 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.604645 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.604647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:28.606225 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:28.606238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.607251 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.607255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:28.607414 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:28.607486 (+ 72us) client_negotiation.cc:769] Sending connection context
0925 22:58:28.607574 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 22:58:28.607694 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":225,"thread_start_us":141,"threads_started":1}
I20260925 22:58:28.608721 5938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426be197457646c496daabf4c177d35a" candidate_uuid: "6c82979c633a46a587cae59eb389a78e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afe83aedf6ac4e5fb36c1ec70d821128" is_pre_election: true
I20260925 22:58:28.609207 5938 raft_consensus.cc:2454] T 426be197457646c496daabf4c177d35a P afe83aedf6ac4e5fb36c1ec70d821128 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c82979c633a46a587cae59eb389a78e in term 0.
I20260925 22:58:28.609512 5670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426be197457646c496daabf4c177d35a" candidate_uuid: "6c82979c633a46a587cae59eb389a78e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6616ff11012a4d99be6ac8b9c5b7e16a" is_pre_election: true
I20260925 22:58:28.609836 5670 raft_consensus.cc:2454] T 426be197457646c496daabf4c177d35a P 6616ff11012a4d99be6ac8b9c5b7e16a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c82979c633a46a587cae59eb389a78e in term 0.
I20260925 22:58:28.609901 5737 leader_election.cc:302] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c82979c633a46a587cae59eb389a78e, afe83aedf6ac4e5fb36c1ec70d821128; no voters:
I20260925 22:58:28.610291 6008 raft_consensus.cc:2781] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:58:28.610427 6008 raft_consensus.cc:482] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:58:28.610491 6008 raft_consensus.cc:3037] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:28.611606 6008 raft_consensus.cc:504] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe83aedf6ac4e5fb36c1ec70d821128" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 37053 } } peers { permanent_uuid: "6c82979c633a46a587cae59eb389a78e" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 38721 } } peers { permanent_uuid: "6616ff11012a4d99be6ac8b9c5b7e16a" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 46075 } }
I20260925 22:58:28.612028 6008 leader_election.cc:288] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e [CANDIDATE]: Term 1 election: Requested vote from peers afe83aedf6ac4e5fb36c1ec70d821128 (127.20.114.195:37053), 6616ff11012a4d99be6ac8b9c5b7e16a (127.20.114.193:46075)
I20260925 22:58:28.612448 5938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426be197457646c496daabf4c177d35a" candidate_uuid: "6c82979c633a46a587cae59eb389a78e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afe83aedf6ac4e5fb36c1ec70d821128"
I20260925 22:58:28.612540 5670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "426be197457646c496daabf4c177d35a" candidate_uuid: "6c82979c633a46a587cae59eb389a78e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6616ff11012a4d99be6ac8b9c5b7e16a"
I20260925 22:58:28.612603 5938 raft_consensus.cc:3037] T 426be197457646c496daabf4c177d35a P afe83aedf6ac4e5fb36c1ec70d821128 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:28.612658 5670 raft_consensus.cc:3037] T 426be197457646c496daabf4c177d35a P 6616ff11012a4d99be6ac8b9c5b7e16a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:28.614151 5938 raft_consensus.cc:2454] T 426be197457646c496daabf4c177d35a P afe83aedf6ac4e5fb36c1ec70d821128 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c82979c633a46a587cae59eb389a78e in term 1.
I20260925 22:58:28.614151 5670 raft_consensus.cc:2454] T 426be197457646c496daabf4c177d35a P 6616ff11012a4d99be6ac8b9c5b7e16a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c82979c633a46a587cae59eb389a78e in term 1.
I20260925 22:58:28.614615 5740 leader_election.cc:302] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6616ff11012a4d99be6ac8b9c5b7e16a, 6c82979c633a46a587cae59eb389a78e; no voters:
I20260925 22:58:28.614871 6008 raft_consensus.cc:2781] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:28.615142 6008 raft_consensus.cc:686] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e [term 1 LEADER]: Becoming Leader. State: Replica: 6c82979c633a46a587cae59eb389a78e, State: Running, Role: LEADER
I20260925 22:58:28.615546 6008 consensus_queue.cc:237] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe83aedf6ac4e5fb36c1ec70d821128" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 37053 } } peers { permanent_uuid: "6c82979c633a46a587cae59eb389a78e" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 38721 } } peers { permanent_uuid: "6616ff11012a4d99be6ac8b9c5b7e16a" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 46075 } }
I20260925 22:58:28.619177 5525 catalog_manager.cc:5495] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c82979c633a46a587cae59eb389a78e (127.20.114.194). New cstate: current_term: 1 leader_uuid: "6c82979c633a46a587cae59eb389a78e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe83aedf6ac4e5fb36c1ec70d821128" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 37053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c82979c633a46a587cae59eb389a78e" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6616ff11012a4d99be6ac8b9c5b7e16a" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 46075 } health_report { overall_health: UNKNOWN } } }
I20260925 22:58:28.652611 5999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:28.648770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52682 (local address 127.20.114.194:38721)
0925 22:58:28.648990 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:28.648994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:28.649008 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:58:28.649086 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:28.649088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:28.649145 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:28.649288 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:58:28.649550 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.650099 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.651065 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.651283 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.651370 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:28.651708 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:58:28.652150 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:58:28.652244 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:28.652335 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 22:58:28.652393 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 22:58:28.659086 5938 raft_consensus.cc:1261] T 426be197457646c496daabf4c177d35a P afe83aedf6ac4e5fb36c1ec70d821128 [term 1 FOLLOWER]: Refusing update from remote peer 6c82979c633a46a587cae59eb389a78e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:58:28.659086 5670 raft_consensus.cc:1261] T 426be197457646c496daabf4c177d35a P 6616ff11012a4d99be6ac8b9c5b7e16a [term 1 FOLLOWER]: Refusing update from remote peer 6c82979c633a46a587cae59eb389a78e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:58:28.659855 6011 consensus_queue.cc:1035] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e [LEADER]: Connected to new peer: Peer: permanent_uuid: "afe83aedf6ac4e5fb36c1ec70d821128" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 37053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:58:28.660058 6008 consensus_queue.cc:1035] T 426be197457646c496daabf4c177d35a P 6c82979c633a46a587cae59eb389a78e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6616ff11012a4d99be6ac8b9c5b7e16a" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 46075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:58:28.668422 6016 mvcc.cc:205] Tried to move back new op lower bound from 7333384637059276800 to 7333384636892381184. Current Snapshot: MvccSnapshot[applied={T|T < 7333384637059276800}]
I20260925 22:58:28.669462 6015 mvcc.cc:205] Tried to move back new op lower bound from 7333384637059276800 to 7333384636892381184. Current Snapshot: MvccSnapshot[applied={T|T < 7333384637059276800}]
I20260925 22:58:28.677266 5522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:58:28.677465 5522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:58:28.680365 5522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9729d15cfec842d79a00b2aaa1184d63: Sending DeleteTablet for 3 replicas of tablet 426be197457646c496daabf4c177d35a
I20260925 22:58:28.681331 5918 tablet_service.cc:1514] Processing DeleteTablet for tablet 426be197457646c496daabf4c177d35a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:58:28 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:39764
I20260925 22:58:28.681385 5650 tablet_service.cc:1514] Processing DeleteTablet for tablet 426be197457646c496daabf4c177d35a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:58:28 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:44838
I20260925 22:58:28.681950 6024 tablet_replica.cc:330] T 426be197457646c496daabf4c177d35a P 6616ff11012a4d99be6ac8b9c5b7e16a: stopping tablet replica
I20260925 22:58:28.682008 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 5587
I20260925 22:58:28.682116 5784 tablet_service.cc:1514] Processing DeleteTablet for tablet 426be197457646c496daabf4c177d35a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:58:28 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:52672
I20260925 22:58:28.682343 6025 tablet_replica.cc:330] T 426be197457646c496daabf4c177d35a P afe83aedf6ac4e5fb36c1ec70d821128: stopping tablet replica
I20260925 22:58:28.682802 6025 raft_consensus.cc:2229] T 426be197457646c496daabf4c177d35a P afe83aedf6ac4e5fb36c1ec70d821128 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:58:28.683241 6025 raft_consensus.cc:2258] T 426be197457646c496daabf4c177d35a P afe83aedf6ac4e5fb36c1ec70d821128 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:58:28.685153 6025 ts_tablet_manager.cc:1905] T 426be197457646c496daabf4c177d35a P afe83aedf6ac4e5fb36c1ec70d821128: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:58:28.688184 6025 ts_tablet_manager.cc:1918] T 426be197457646c496daabf4c177d35a P afe83aedf6ac4e5fb36c1ec70d821128: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:58:28.688328 6025 log.cc:1196] T 426be197457646c496daabf4c177d35a P afe83aedf6ac4e5fb36c1ec70d821128: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/426be197457646c496daabf4c177d35a
I20260925 22:58:28.688673 6025 ts_tablet_manager.cc:1939] T 426be197457646c496daabf4c177d35a P afe83aedf6ac4e5fb36c1ec70d821128: Deleting consensus metadata
W20260925 22:58:28.689489 5512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.114.193:46075 (error 108)
I20260925 22:58:28.690153 5509 catalog_manager.cc:4841] TS afe83aedf6ac4e5fb36c1ec70d821128 (127.20.114.195:37053): tablet 426be197457646c496daabf4c177d35a (table test-table [id=375e42099779424d94fe45df9abece4b]) successfully deleted
I20260925 22:58:28.690642 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 5721
I20260925 22:58:28.698690 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 5855
I20260925 22:58:28.701057 5997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:28.700646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:46075 (local address 127.0.0.1:44842)
0925 22:58:28.700836 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:28.700958 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.193:46075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 22:58:28.701299 5998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:28.700948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:38721 (local address 127.0.0.1:52696)
0925 22:58:28.701131 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:28.701190 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.194:38721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260925 22:58:28.701378 5512 catalog_manager.cc:4584] TS 6616ff11012a4d99be6ac8b9c5b7e16a (127.20.114.193:46075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 426be197457646c496daabf4c177d35a: Network error: Client connection negotiation failed: client connection to 127.20.114.193:46075: connect: Connection refused (error 111)
I20260925 22:58:28.704977 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 5493
2026-09-25T22:58:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2438 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T22:58:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:58:28Z Disabled control of system clock
I20260925 22:58:28.736706 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:45721
--webserver_interface=127.20.114.254
--webserver_port=0
--builtin_ntp_servers=127.20.114.212:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:45721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:58:28.855706 6031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:28.856041 6031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:28.856082 6031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:28.860430 6031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:58:28.860520 6031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:28.860536 6031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:28.860554 6031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:58:28.860571 6031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:58:28.865934 6031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:45721
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:45721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:28.867264 6031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:28.868777 6031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:28.877969 6039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:28.877990 6036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:28.877957 6037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:28.878284 6031 server_base.cc:1034] running on GCE node
I20260925 22:58:28.879066 6031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:28.880311 6031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:28.881520 6031 hybrid_clock.cc:648] HybridClock initialized: now 1790377108881501 us; error 34 us; skew 500 ppm
I20260925 22:58:28.883631 6031 webserver.cc:460] Webserver started at http://127.20.114.254:43563/ using document root <none> and password file <none>
I20260925 22:58:28.884262 6031 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:28.884331 6031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:28.884560 6031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:28.886754 6031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "f7f469704e86423d85c949911eb6f916"
format_stamp: "Formatted at 2026-09-25 22:58:28 on dist-test-slave-x1lh"
server_key: "410db9c199d94f8c62cfb07a9a1e610d"
server_key_iv: "3f22e3bc78880ab6a1db32050744ea7a"
server_key_version: "encryptionkey@0"
I20260925 22:58:28.887364 6031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "f7f469704e86423d85c949911eb6f916"
format_stamp: "Formatted at 2026-09-25 22:58:28 on dist-test-slave-x1lh"
server_key: "410db9c199d94f8c62cfb07a9a1e610d"
server_key_iv: "3f22e3bc78880ab6a1db32050744ea7a"
server_key_version: "encryptionkey@0"
I20260925 22:58:28.891530 6031 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:58:28.894466 6045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:28.896070 6031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:58:28.896256 6031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "f7f469704e86423d85c949911eb6f916"
format_stamp: "Formatted at 2026-09-25 22:58:28 on dist-test-slave-x1lh"
server_key: "410db9c199d94f8c62cfb07a9a1e610d"
server_key_iv: "3f22e3bc78880ab6a1db32050744ea7a"
server_key_version: "encryptionkey@0"
I20260925 22:58:28.896425 6031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:28.917282 6031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:28.918152 6031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:28.918391 6031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:28.925966 6031 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:45721
I20260925 22:58:28.925976 6097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:45721 every 8 connection(s)
I20260925 22:58:28.927040 6031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:58:28.930557 6098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:28.933981 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 6031
I20260925 22:58:28.934146 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:58:28.934464 20939 external_mini_cluster.cc:1428] Setting key 6b2793ebb3f365a648e59a50b0344b27
I20260925 22:58:28.936805 6098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916: Bootstrap starting.
I20260925 22:58:28.939399 6098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:28.940378 6098 log.cc:824] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:28.943156 6098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916: No bootstrap required, opened a new log
I20260925 22:58:28.944856 6101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:28.936025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41404 (local address 127.20.114.254:45721)
0925 22:58:28.936700 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:28.936724 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:28.936761 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:58:28.937528 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:28.937546 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:28.937978 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:28.938279 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:28.938672 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.939760 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.941053 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:28.941722 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:28.941916 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:28.942861 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:28.942867 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:28.942886 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:28.943387 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:28.943458 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:28.943481 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:28.943565 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:28.943741 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 22:58:28.944077 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":160,"threads_started":1}
I20260925 22:58:28.947518 6098 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f469704e86423d85c949911eb6f916" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45721 } }
I20260925 22:58:28.947854 6098 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:28.947943 6098 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f469704e86423d85c949911eb6f916, State: Initialized, Role: FOLLOWER
I20260925 22:58:28.948393 6098 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f469704e86423d85c949911eb6f916" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45721 } }
I20260925 22:58:28.948547 6098 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:58:28.948617 6098 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:58:28.948748 6098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:28.949966 6098 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f469704e86423d85c949911eb6f916" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45721 } }
I20260925 22:58:28.950340 6098 leader_election.cc:302] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7f469704e86423d85c949911eb6f916; no voters:
I20260925 22:58:28.950685 6098 leader_election.cc:288] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:58:28.950911 6103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:28.951149 6103 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916 [term 1 LEADER]: Becoming Leader. State: Replica: f7f469704e86423d85c949911eb6f916, State: Running, Role: LEADER
I20260925 22:58:28.951506 6103 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f469704e86423d85c949911eb6f916" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45721 } }
I20260925 22:58:28.952041 6098 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:58:28.954286 6104 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7f469704e86423d85c949911eb6f916" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f469704e86423d85c949911eb6f916" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45721 } } }
I20260925 22:58:28.954283 6105 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7f469704e86423d85c949911eb6f916. Latest consensus state: current_term: 1 leader_uuid: "f7f469704e86423d85c949911eb6f916" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f469704e86423d85c949911eb6f916" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45721 } } }
I20260925 22:58:28.954797 6104 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:28.954797 6105 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916 [sys.catalog]: This master's current role is: LEADER
W20260925 22:58:28.957180 6117 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:58:28.957248 6117 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:58:28.957335 6119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:58:28.958045 6119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:58:28.964077 6119 catalog_manager.cc:1347] Generated new cluster ID: 16ee0a81016b4598be5011422e2f7d79
I20260925 22:58:28.964167 6119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:58:28.976627 6119 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:58:28.977771 6119 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:58:28.988231 6119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916: Generated new TSK 0
I20260925 22:58:28.989233 6119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:58:28.997349 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:45721
--builtin_ntp_servers=127.20.114.212:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:58:29.116098 6122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:29.116451 6122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:29.116541 6122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:29.120827 6122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:29.120961 6122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:29.121061 6122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:58:29.126134 6122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:45721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:29.127338 6122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:29.128760 6122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:29.138751 6127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:29.138767 6130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:29.138818 6122 server_base.cc:1034] running on GCE node
W20260925 22:58:29.138767 6128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:29.139483 6122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:29.140233 6122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:29.141426 6122 hybrid_clock.cc:648] HybridClock initialized: now 1790377109141409 us; error 36 us; skew 500 ppm
I20260925 22:58:29.143478 6122 webserver.cc:460] Webserver started at http://127.20.114.193:39565/ using document root <none> and password file <none>
I20260925 22:58:29.144097 6122 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:29.144196 6122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:29.144434 6122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:29.146704 6122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "ee4dd8e473dd4980bfee939bf1ccbb82"
format_stamp: "Formatted at 2026-09-25 22:58:29 on dist-test-slave-x1lh"
server_key: "02d7f55edbf7489284a96dcb0eab1cbd"
server_key_iv: "541fecb4781fe5b605fe2249cfaa7328"
server_key_version: "encryptionkey@0"
I20260925 22:58:29.147308 6122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "ee4dd8e473dd4980bfee939bf1ccbb82"
format_stamp: "Formatted at 2026-09-25 22:58:29 on dist-test-slave-x1lh"
server_key: "02d7f55edbf7489284a96dcb0eab1cbd"
server_key_iv: "541fecb4781fe5b605fe2249cfaa7328"
server_key_version: "encryptionkey@0"
I20260925 22:58:29.151324 6122 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 22:58:29.154352 6136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:29.155817 6122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:58:29.155941 6122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "ee4dd8e473dd4980bfee939bf1ccbb82"
format_stamp: "Formatted at 2026-09-25 22:58:29 on dist-test-slave-x1lh"
server_key: "02d7f55edbf7489284a96dcb0eab1cbd"
server_key_iv: "541fecb4781fe5b605fe2249cfaa7328"
server_key_version: "encryptionkey@0"
I20260925 22:58:29.156049 6122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:29.175213 6122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:29.175904 6122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:29.176061 6122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:29.176597 6122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:58:29.177603 6122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:58:29.177652 6122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:29.177693 6122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:58:29.177708 6122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:29.187065 6122 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:44021
I20260925 22:58:29.187096 6249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:44021 every 8 connection(s)
I20260925 22:58:29.188072 6122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:58:29.194963 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 6122
I20260925 22:58:29.195130 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:58:29.195453 20939 external_mini_cluster.cc:1428] Setting key 28fddf74f1dd62b8ae8347e124813697
I20260925 22:58:29.197846 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:45721
--builtin_ntp_servers=127.20.114.212:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:58:29.199550 6101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:29.190219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:44019 (local address 127.20.114.254:45721)
0925 22:58:29.190390 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:29.190394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:29.191662 (+ 1268us) server_negotiation.cc:407] Connection header received
0925 22:58:29.193251 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:29.193254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:29.193323 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:29.193423 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:29.194607 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.195297 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.196905 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.197207 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.197307 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:29.198862 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:29.198866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:29.198868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:29.198883 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:29.198906 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:29.198916 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:29.199011 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:29.199250 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 22:58:29.199333 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 22:58:29.200589 6252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:29.190667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:45721 (local address 127.20.114.193:44019)
0925 22:58:29.191456 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:29.191503 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:29.192972 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:29.193708 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:29.193716 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:29.193823 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:58:29.194428 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:29.194442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.195452 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.195456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:29.196697 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:29.196709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.197408 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.197413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:29.197666 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:29.198431 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:58:29.198435 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:58:29.198501 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:58:29.198601 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:58:29.199036 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:58:29.199049 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:58:29.199059 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 22:58:29.199207 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 22:58:29.199557 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":130,"threads_started":1}
I20260925 22:58:29.202112 6250 heartbeater.cc:347] Connected to a master server at 127.20.114.254:45721
I20260925 22:58:29.202414 6250 heartbeater.cc:464] Registering TS with master...
I20260925 22:58:29.203003 6250 heartbeater.cc:511] Master 127.20.114.254:45721 requested a full tablet report, sending...
I20260925 22:58:29.204929 6062 ts_manager.cc:194] Registered new tserver with Master: ee4dd8e473dd4980bfee939bf1ccbb82 (127.20.114.193:44021)
I20260925 22:58:29.206246 6062 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.114.193:44019
W20260925 22:58:29.324393 6253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:29.324750 6253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:29.324834 6253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:29.329422 6253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:29.329509 6253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:29.329573 6253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:58:29.335136 6253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:45721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:29.336246 6253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:29.337662 6253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:58:29.348084 6253 server_base.cc:1034] running on GCE node
W20260925 22:58:29.348053 6258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:29.348083 6259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:29.348402 6261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:29.348929 6253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:29.349704 6253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:29.350942 6253 hybrid_clock.cc:648] HybridClock initialized: now 1790377109350897 us; error 51 us; skew 500 ppm
I20260925 22:58:29.353377 6253 webserver.cc:460] Webserver started at http://127.20.114.194:46515/ using document root <none> and password file <none>
I20260925 22:58:29.354071 6253 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:29.354168 6253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:29.354395 6253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:29.356600 6253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "cf0479134c2745adb1020f8ac5b134d8"
format_stamp: "Formatted at 2026-09-25 22:58:29 on dist-test-slave-x1lh"
server_key: "d6d1b2fbfb50eacdbfcbaa4dc9a309c2"
server_key_iv: "97746b4cb85923f78da093297a19f511"
server_key_version: "encryptionkey@0"
I20260925 22:58:29.357270 6253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "cf0479134c2745adb1020f8ac5b134d8"
format_stamp: "Formatted at 2026-09-25 22:58:29 on dist-test-slave-x1lh"
server_key: "d6d1b2fbfb50eacdbfcbaa4dc9a309c2"
server_key_iv: "97746b4cb85923f78da093297a19f511"
server_key_version: "encryptionkey@0"
I20260925 22:58:29.361935 6253 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:58:29.365913 6267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:29.368177 6253 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:58:29.368343 6253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "cf0479134c2745adb1020f8ac5b134d8"
format_stamp: "Formatted at 2026-09-25 22:58:29 on dist-test-slave-x1lh"
server_key: "d6d1b2fbfb50eacdbfcbaa4dc9a309c2"
server_key_iv: "97746b4cb85923f78da093297a19f511"
server_key_version: "encryptionkey@0"
I20260925 22:58:29.368499 6253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:29.387135 6253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:29.387933 6253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:29.388155 6253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:29.388788 6253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:58:29.389851 6253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:58:29.389926 6253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:29.390002 6253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:58:29.390053 6253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:29.399977 6253 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:37555
I20260925 22:58:29.400075 6380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:37555 every 8 connection(s)
I20260925 22:58:29.401247 6253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:58:29.406291 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 6253
I20260925 22:58:29.406416 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:58:29.406764 20939 external_mini_cluster.cc:1428] Setting key fcfb98d1d17ac0e795e18067e38923e8
I20260925 22:58:29.409153 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:45721
--builtin_ntp_servers=127.20.114.212:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:58:29.413062 6101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:29.403443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:40669 (local address 127.20.114.254:45721)
0925 22:58:29.403611 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:29.403617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:29.405026 (+ 1409us) server_negotiation.cc:407] Connection header received
0925 22:58:29.406735 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:29.406739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:29.406819 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:29.406943 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:29.408314 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.408956 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.410607 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.410946 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.411017 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:29.412473 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:29.412477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:29.412479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:29.412495 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:29.412514 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:29.412522 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:29.412603 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:29.412795 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 22:58:29.412905 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:58:29.414155 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:29.403940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:45721 (local address 127.20.114.194:40669)
0925 22:58:29.404703 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:29.404755 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:29.406444 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:29.407220 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:29.407232 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:29.407409 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:58:29.408037 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:29.408055 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.409310 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.409314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:29.410419 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:29.410438 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.411206 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.411208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:29.411389 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:29.412130 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:58:29.412132 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:58:29.412182 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:58:29.412274 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:58:29.412619 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:58:29.412632 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:58:29.412642 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 22:58:29.412780 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 22:58:29.413146 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":129,"threads_started":1}
I20260925 22:58:29.415622 6381 heartbeater.cc:347] Connected to a master server at 127.20.114.254:45721
I20260925 22:58:29.415964 6381 heartbeater.cc:464] Registering TS with master...
I20260925 22:58:29.416587 6381 heartbeater.cc:511] Master 127.20.114.254:45721 requested a full tablet report, sending...
I20260925 22:58:29.417959 6062 ts_manager.cc:194] Registered new tserver with Master: cf0479134c2745adb1020f8ac5b134d8 (127.20.114.194:37555)
I20260925 22:58:29.418891 6062 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.114.194:40669
W20260925 22:58:29.531350 6384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:29.531647 6384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:29.531688 6384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:29.536158 6384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:29.536232 6384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:29.536294 6384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:58:29.541520 6384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:45721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:29.542666 6384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:29.544252 6384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:58:29.554564 6384 server_base.cc:1034] running on GCE node
W20260925 22:58:29.554481 6389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:29.554466 6392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:29.554461 6390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:29.555434 6384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:29.556156 6384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:29.557367 6384 hybrid_clock.cc:648] HybridClock initialized: now 1790377109557352 us; error 33 us; skew 500 ppm
I20260925 22:58:29.565469 6384 webserver.cc:460] Webserver started at http://127.20.114.195:39405/ using document root <none> and password file <none>
I20260925 22:58:29.566144 6384 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:29.566262 6384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:29.566514 6384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:29.568760 6384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "79d63aca23e341b1bd945ea11be8669c"
format_stamp: "Formatted at 2026-09-25 22:58:29 on dist-test-slave-x1lh"
server_key: "132b5a76507e82e3de6787333150a450"
server_key_iv: "93267fe1a763127dfdc0e93a5b51d017"
server_key_version: "encryptionkey@0"
I20260925 22:58:29.569473 6384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "79d63aca23e341b1bd945ea11be8669c"
format_stamp: "Formatted at 2026-09-25 22:58:29 on dist-test-slave-x1lh"
server_key: "132b5a76507e82e3de6787333150a450"
server_key_iv: "93267fe1a763127dfdc0e93a5b51d017"
server_key_version: "encryptionkey@0"
I20260925 22:58:29.573474 6384 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:58:29.576426 6398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:29.578033 6384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:29.578166 6384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "79d63aca23e341b1bd945ea11be8669c"
format_stamp: "Formatted at 2026-09-25 22:58:29 on dist-test-slave-x1lh"
server_key: "132b5a76507e82e3de6787333150a450"
server_key_iv: "93267fe1a763127dfdc0e93a5b51d017"
server_key_version: "encryptionkey@0"
I20260925 22:58:29.578279 6384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:29.595198 6384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:29.595916 6384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:29.596082 6384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:29.596752 6384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:58:29.597841 6384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:58:29.597888 6384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:29.597929 6384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:58:29.597945 6384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:29.607882 6384 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:42503
I20260925 22:58:29.607916 6511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:42503 every 8 connection(s)
I20260925 22:58:29.608956 6384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:58:29.616984 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 6384
I20260925 22:58:29.617142 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:58:29.617422 20939 external_mini_cluster.cc:1428] Setting key 3901705c7a54a8c9f44dad191b7a8e7a
I20260925 22:58:29.619684 6101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:29.611087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:58403 (local address 127.20.114.254:45721)
0925 22:58:29.611239 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:29.611244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:29.612513 (+ 1269us) server_negotiation.cc:407] Connection header received
0925 22:58:29.614121 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:29.614125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:29.614193 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:29.614276 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:29.615291 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.615848 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.617208 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.617454 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.617592 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:29.619071 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:29.619074 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:29.619076 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:29.619092 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:29.619114 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:29.619122 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:29.619219 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:29.619459 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 22:58:29.619536 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:58:29.620748 6514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:29.611517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:45721 (local address 127.20.114.195:58403)
0925 22:58:29.612293 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:29.612342 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:29.613851 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:29.614479 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:29.614487 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:29.614586 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:58:29.615125 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:29.615140 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.616012 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.616015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:29.617065 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:29.617077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.617645 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.617649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:29.617891 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:29.618687 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:58:29.618690 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:58:29.618753 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:58:29.618857 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:58:29.619221 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:58:29.619236 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:58:29.619246 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 22:58:29.619450 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 22:58:29.619807 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":134,"threads_started":1}
I20260925 22:58:29.622148 6512 heartbeater.cc:347] Connected to a master server at 127.20.114.254:45721
I20260925 22:58:29.622430 6512 heartbeater.cc:464] Registering TS with master...
I20260925 22:58:29.622997 6512 heartbeater.cc:511] Master 127.20.114.254:45721 requested a full tablet report, sending...
I20260925 22:58:29.624112 6062 ts_manager.cc:194] Registered new tserver with Master: 79d63aca23e341b1bd945ea11be8669c (127.20.114.195:42503)
I20260925 22:58:29.624704 6062 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.114.195:58403
I20260925 22:58:29.632388 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:58:29.638763 6101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:29.634602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41418 (local address 127.20.114.254:45721)
0925 22:58:29.634801 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:29.634806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:29.634991 (+ 185us) server_negotiation.cc:407] Connection header received
0925 22:58:29.635158 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:29.635165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:29.635210 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:29.635300 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:29.635603 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.636134 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.637076 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.637310 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.637425 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:29.638313 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:29.638328 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:29.638330 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:29.638344 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:29.638363 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:29.638371 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:29.638453 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:29.638544 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 22:58:29.638608 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 22:58:29.642370 6062 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:58:29.645651 6062 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:58:29.662184 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:29.655716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:42503 (local address 127.0.0.1:55098)
0925 22:58:29.656371 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:29.656438 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:29.656591 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:29.657334 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:29.657341 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:29.657366 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:29.657548 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:29.657556 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.658888 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.658893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:29.660321 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:29.660335 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.661302 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.661306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:29.661678 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:29.661747 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 22:58:29.661872 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 22:58:29.662011 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":173,"thread_start_us":104,"threads_started":1}
I20260925 22:58:29.662258 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:29.656018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55098 (local address 127.20.114.195:42503)
0925 22:58:29.656537 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:29.656550 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:29.656580 (+ 30us) server_negotiation.cc:407] Connection header received
0925 22:58:29.656817 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:29.656824 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:29.657129 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:29.657316 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:29.657686 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.658777 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.660520 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.661174 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.661317 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:29.661544 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:29.661944 (+ 400us) server_negotiation.cc:299] Negotiation successful
0925 22:58:29.662073 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":174,"threads_started":1}
I20260925 22:58:29.663720 6525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:29.656105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:44021 (local address 127.0.0.1:35602)
0925 22:58:29.656843 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:29.656897 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:29.657035 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:29.658294 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:29.658297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:29.658315 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:29.658410 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:29.658416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.659911 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.659914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:29.661635 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:29.661648 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.663265 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.663268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:29.663364 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:29.663387 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:58:29.663489 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 22:58:29.663569 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":252,"thread_start_us":105,"threads_started":1}
I20260925 22:58:29.663784 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:29.656951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35602 (local address 127.20.114.193:44021)
0925 22:58:29.657714 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:29.657721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:29.657746 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:58:29.657819 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:29.657825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:29.658015 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:29.658157 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:29.658552 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.659783 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.661778 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.662897 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.663018 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:29.663201 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:29.663525 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 22:58:29.663629 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":118,"threads_started":1}
I20260925 22:58:29.665670 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:29.656497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:37555 (local address 127.0.0.1:39398)
0925 22:58:29.657377 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:29.657398 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:29.657476 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:29.658032 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:29.658036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:29.658055 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:29.658168 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:29.658177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.659839 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.659845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:29.661656 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:29.661662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.665226 (+ 3564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.665229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:29.665307 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:29.665326 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 22:58:29.665441 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 22:58:29.665525 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":215,"thread_start_us":83,"threads_started":1}
I20260925 22:58:29.665763 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:29.657255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39398 (local address 127.20.114.194:37555)
0925 22:58:29.657635 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:29.657645 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:29.657667 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:58:29.657731 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:29.657734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:29.657882 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:29.658054 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:29.658303 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.659688 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.661809 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.663101 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.663237 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:29.665162 (+ 1925us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:29.665490 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 22:58:29.665603 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":145,"threads_started":1}
I20260925 22:58:29.665792 6446 tablet_service.cc:1467] Processing CreateTablet for tablet 98316ec5887e4a0ca8ab7698fa4d4cf0 (DEFAULT_TABLE table=test-table [id=0542968351b94c4aba132045f3a3501e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:58:29.667114 6446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98316ec5887e4a0ca8ab7698fa4d4cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:29.667004 6184 tablet_service.cc:1467] Processing CreateTablet for tablet 98316ec5887e4a0ca8ab7698fa4d4cf0 (DEFAULT_TABLE table=test-table [id=0542968351b94c4aba132045f3a3501e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:58:29.668197 6184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98316ec5887e4a0ca8ab7698fa4d4cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:29.672060 6315 tablet_service.cc:1467] Processing CreateTablet for tablet 98316ec5887e4a0ca8ab7698fa4d4cf0 (DEFAULT_TABLE table=test-table [id=0542968351b94c4aba132045f3a3501e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:58:29.673259 6315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98316ec5887e4a0ca8ab7698fa4d4cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:29.675771 6529 tablet_bootstrap.cc:492] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c: Bootstrap starting.
I20260925 22:58:29.677196 6530 tablet_bootstrap.cc:492] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P ee4dd8e473dd4980bfee939bf1ccbb82: Bootstrap starting.
I20260925 22:58:29.678647 6529 tablet_bootstrap.cc:654] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:29.679409 6530 tablet_bootstrap.cc:654] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P ee4dd8e473dd4980bfee939bf1ccbb82: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:29.679659 6529 log.cc:824] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:29.680300 6530 log.cc:824] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P ee4dd8e473dd4980bfee939bf1ccbb82: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:29.680575 6532 tablet_bootstrap.cc:492] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P cf0479134c2745adb1020f8ac5b134d8: Bootstrap starting.
I20260925 22:58:29.682284 6529 tablet_bootstrap.cc:492] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c: No bootstrap required, opened a new log
I20260925 22:58:29.682416 6530 tablet_bootstrap.cc:492] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P ee4dd8e473dd4980bfee939bf1ccbb82: No bootstrap required, opened a new log
I20260925 22:58:29.682529 6529 ts_tablet_manager.cc:1397] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20260925 22:58:29.682651 6530 ts_tablet_manager.cc:1397] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P ee4dd8e473dd4980bfee939bf1ccbb82: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:58:29.683555 6532 tablet_bootstrap.cc:654] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P cf0479134c2745adb1020f8ac5b134d8: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:29.684437 6532 log.cc:824] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P cf0479134c2745adb1020f8ac5b134d8: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:29.686760 6532 tablet_bootstrap.cc:492] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P cf0479134c2745adb1020f8ac5b134d8: No bootstrap required, opened a new log
I20260925 22:58:29.686971 6532 ts_tablet_manager.cc:1397] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P cf0479134c2745adb1020f8ac5b134d8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 22:58:29.686880 6530 raft_consensus.cc:348] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P ee4dd8e473dd4980bfee939bf1ccbb82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d63aca23e341b1bd945ea11be8669c" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42503 } } peers { permanent_uuid: "cf0479134c2745adb1020f8ac5b134d8" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 37555 } } peers { permanent_uuid: "ee4dd8e473dd4980bfee939bf1ccbb82" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44021 } }
I20260925 22:58:29.686970 6529 raft_consensus.cc:348] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d63aca23e341b1bd945ea11be8669c" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42503 } } peers { permanent_uuid: "cf0479134c2745adb1020f8ac5b134d8" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 37555 } } peers { permanent_uuid: "ee4dd8e473dd4980bfee939bf1ccbb82" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44021 } }
I20260925 22:58:29.687213 6530 raft_consensus.cc:374] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P ee4dd8e473dd4980bfee939bf1ccbb82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:29.687264 6529 raft_consensus.cc:374] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:29.687285 6530 raft_consensus.cc:729] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P ee4dd8e473dd4980bfee939bf1ccbb82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4dd8e473dd4980bfee939bf1ccbb82, State: Initialized, Role: FOLLOWER
I20260925 22:58:29.687331 6529 raft_consensus.cc:729] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79d63aca23e341b1bd945ea11be8669c, State: Initialized, Role: FOLLOWER
I20260925 22:58:29.687815 6530 consensus_queue.cc:260] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P ee4dd8e473dd4980bfee939bf1ccbb82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d63aca23e341b1bd945ea11be8669c" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42503 } } peers { permanent_uuid: "cf0479134c2745adb1020f8ac5b134d8" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 37555 } } peers { permanent_uuid: "ee4dd8e473dd4980bfee939bf1ccbb82" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44021 } }
I20260925 22:58:29.687878 6529 consensus_queue.cc:260] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d63aca23e341b1bd945ea11be8669c" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42503 } } peers { permanent_uuid: "cf0479134c2745adb1020f8ac5b134d8" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 37555 } } peers { permanent_uuid: "ee4dd8e473dd4980bfee939bf1ccbb82" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44021 } }
I20260925 22:58:29.688603 6250 heartbeater.cc:503] Master 127.20.114.254:45721 was elected leader, sending a full tablet report...
I20260925 22:58:29.688992 6529 ts_tablet_manager.cc:1428] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260925 22:58:29.688993 6530 ts_tablet_manager.cc:1428] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P ee4dd8e473dd4980bfee939bf1ccbb82: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260925 22:58:29.689267 6251 tablet.cc:2369] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P ee4dd8e473dd4980bfee939bf1ccbb82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:58:29.689337 6512 heartbeater.cc:503] Master 127.20.114.254:45721 was elected leader, sending a full tablet report...
I20260925 22:58:29.691852 6532 raft_consensus.cc:348] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P cf0479134c2745adb1020f8ac5b134d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d63aca23e341b1bd945ea11be8669c" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42503 } } peers { permanent_uuid: "cf0479134c2745adb1020f8ac5b134d8" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 37555 } } peers { permanent_uuid: "ee4dd8e473dd4980bfee939bf1ccbb82" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44021 } }
I20260925 22:58:29.692154 6532 raft_consensus.cc:374] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P cf0479134c2745adb1020f8ac5b134d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:29.692219 6532 raft_consensus.cc:729] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P cf0479134c2745adb1020f8ac5b134d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf0479134c2745adb1020f8ac5b134d8, State: Initialized, Role: FOLLOWER
I20260925 22:58:29.692817 6532 consensus_queue.cc:260] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P cf0479134c2745adb1020f8ac5b134d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d63aca23e341b1bd945ea11be8669c" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42503 } } peers { permanent_uuid: "cf0479134c2745adb1020f8ac5b134d8" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 37555 } } peers { permanent_uuid: "ee4dd8e473dd4980bfee939bf1ccbb82" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44021 } }
I20260925 22:58:29.694070 6532 ts_tablet_manager.cc:1428] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P cf0479134c2745adb1020f8ac5b134d8: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260925 22:58:29.694211 6381 heartbeater.cc:503] Master 127.20.114.254:45721 was elected leader, sending a full tablet report...
I20260925 22:58:29.711155 6536 raft_consensus.cc:482] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:58:29.711426 6536 raft_consensus.cc:504] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d63aca23e341b1bd945ea11be8669c" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42503 } } peers { permanent_uuid: "cf0479134c2745adb1020f8ac5b134d8" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 37555 } } peers { permanent_uuid: "ee4dd8e473dd4980bfee939bf1ccbb82" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44021 } }
I20260925 22:58:29.712721 6536 leader_election.cc:288] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf0479134c2745adb1020f8ac5b134d8 (127.20.114.194:37555), ee4dd8e473dd4980bfee939bf1ccbb82 (127.20.114.193:44021)
I20260925 22:58:29.716799 6514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:29.713040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:37555 (local address 127.20.114.195:53291)
0925 22:58:29.713193 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:29.713211 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:29.713326 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:29.713621 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:29.713624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:29.713642 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:29.713805 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:29.713813 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.714714 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.714717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:29.715614 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:29.715623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.716361 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.716366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:29.716468 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:29.716504 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 22:58:29.716579 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 22:58:29.716661 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 22:58:29.716830 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:29.713147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:53291 (local address 127.20.114.194:37555)
0925 22:58:29.713326 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:29.713330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:29.713353 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:58:29.713412 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:29.713415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:29.713496 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:29.713583 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:29.713953 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.714587 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.715754 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.716232 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.716335 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:29.716375 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:29.716616 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 22:58:29.716699 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 22:58:29.717293 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:29.713165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:52157 (local address 127.20.114.193:44021)
0925 22:58:29.713293 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:29.713297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:29.713705 (+ 408us) server_negotiation.cc:407] Connection header received
0925 22:58:29.713813 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:29.713816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:29.713877 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:29.713960 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:29.714254 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.714844 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.715972 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.716456 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.716589 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:29.716632 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:29.716987 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 22:58:29.717076 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 22:58:29.717666 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:29.713040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:44021 (local address 127.20.114.195:52157)
0925 22:58:29.713605 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:29.713619 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:29.713725 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:29.713984 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:29.713987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:29.714005 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:29.714115 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:29.714121 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.714965 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.714967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:29.715841 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:29.715851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.716602 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.716606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:29.716703 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:29.716725 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:58:29.717453 (+ 728us) client_negotiation.cc:240] Negotiation successful
0925 22:58:29.717549 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":185,"thread_start_us":120,"threads_started":1}
I20260925 22:58:29.717865 6335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98316ec5887e4a0ca8ab7698fa4d4cf0" candidate_uuid: "79d63aca23e341b1bd945ea11be8669c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf0479134c2745adb1020f8ac5b134d8" is_pre_election: true
I20260925 22:58:29.718248 6335 raft_consensus.cc:2454] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P cf0479134c2745adb1020f8ac5b134d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79d63aca23e341b1bd945ea11be8669c in term 0.
I20260925 22:58:29.718300 6204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98316ec5887e4a0ca8ab7698fa4d4cf0" candidate_uuid: "79d63aca23e341b1bd945ea11be8669c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4dd8e473dd4980bfee939bf1ccbb82" is_pre_election: true
I20260925 22:58:29.718667 6204 raft_consensus.cc:2454] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P ee4dd8e473dd4980bfee939bf1ccbb82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79d63aca23e341b1bd945ea11be8669c in term 0.
I20260925 22:58:29.718999 6400 leader_election.cc:302] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79d63aca23e341b1bd945ea11be8669c, cf0479134c2745adb1020f8ac5b134d8; no voters:
I20260925 22:58:29.719349 6536 raft_consensus.cc:2781] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:58:29.719444 6536 raft_consensus.cc:482] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:58:29.719478 6536 raft_consensus.cc:3037] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:29.720638 6536 raft_consensus.cc:504] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d63aca23e341b1bd945ea11be8669c" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42503 } } peers { permanent_uuid: "cf0479134c2745adb1020f8ac5b134d8" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 37555 } } peers { permanent_uuid: "ee4dd8e473dd4980bfee939bf1ccbb82" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44021 } }
I20260925 22:58:29.721189 6536 leader_election.cc:288] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c [CANDIDATE]: Term 1 election: Requested vote from peers cf0479134c2745adb1020f8ac5b134d8 (127.20.114.194:37555), ee4dd8e473dd4980bfee939bf1ccbb82 (127.20.114.193:44021)
I20260925 22:58:29.721622 6204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98316ec5887e4a0ca8ab7698fa4d4cf0" candidate_uuid: "79d63aca23e341b1bd945ea11be8669c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4dd8e473dd4980bfee939bf1ccbb82"
I20260925 22:58:29.721688 6335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98316ec5887e4a0ca8ab7698fa4d4cf0" candidate_uuid: "79d63aca23e341b1bd945ea11be8669c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf0479134c2745adb1020f8ac5b134d8"
I20260925 22:58:29.721796 6204 raft_consensus.cc:3037] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P ee4dd8e473dd4980bfee939bf1ccbb82 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:29.721958 6335 raft_consensus.cc:3037] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P cf0479134c2745adb1020f8ac5b134d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:29.723037 6204 raft_consensus.cc:2454] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P ee4dd8e473dd4980bfee939bf1ccbb82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79d63aca23e341b1bd945ea11be8669c in term 1.
I20260925 22:58:29.723076 6335 raft_consensus.cc:2454] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P cf0479134c2745adb1020f8ac5b134d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79d63aca23e341b1bd945ea11be8669c in term 1.
I20260925 22:58:29.723446 6402 leader_election.cc:302] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79d63aca23e341b1bd945ea11be8669c, ee4dd8e473dd4980bfee939bf1ccbb82; no voters:
I20260925 22:58:29.723651 6536 raft_consensus.cc:2781] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:29.723860 6536 raft_consensus.cc:686] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c [term 1 LEADER]: Becoming Leader. State: Replica: 79d63aca23e341b1bd945ea11be8669c, State: Running, Role: LEADER
I20260925 22:58:29.724248 6536 consensus_queue.cc:237] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d63aca23e341b1bd945ea11be8669c" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42503 } } peers { permanent_uuid: "cf0479134c2745adb1020f8ac5b134d8" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 37555 } } peers { permanent_uuid: "ee4dd8e473dd4980bfee939bf1ccbb82" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44021 } }
I20260925 22:58:29.727691 6062 catalog_manager.cc:5495] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c reported cstate change: term changed from 0 to 1, leader changed from <none> to 79d63aca23e341b1bd945ea11be8669c (127.20.114.195). New cstate: current_term: 1 leader_uuid: "79d63aca23e341b1bd945ea11be8669c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d63aca23e341b1bd945ea11be8669c" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 42503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf0479134c2745adb1020f8ac5b134d8" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 37555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee4dd8e473dd4980bfee939bf1ccbb82" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44021 } health_report { overall_health: UNKNOWN } } }
I20260925 22:58:29.743605 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:29.739723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55110 (local address 127.20.114.195:42503)
0925 22:58:29.739841 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:29.739845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:29.739858 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:58:29.740021 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:29.740023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:29.740081 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:29.740174 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:58:29.740505 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.741116 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.742085 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:29.742313 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:29.742400 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:29.742728 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:58:29.743190 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:58:29.743283 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:29.743387 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 22:58:29.743464 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:58:29.750278 6204 raft_consensus.cc:1261] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P ee4dd8e473dd4980bfee939bf1ccbb82 [term 1 FOLLOWER]: Refusing update from remote peer 79d63aca23e341b1bd945ea11be8669c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:58:29.750281 6335 raft_consensus.cc:1261] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P cf0479134c2745adb1020f8ac5b134d8 [term 1 FOLLOWER]: Refusing update from remote peer 79d63aca23e341b1bd945ea11be8669c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:58:29.751127 6536 consensus_queue.cc:1035] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf0479134c2745adb1020f8ac5b134d8" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 37555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:58:29.751375 6539 consensus_queue.cc:1035] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4dd8e473dd4980bfee939bf1ccbb82" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 44021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:58:29.763690 6543 mvcc.cc:205] Tried to move back new op lower bound from 7333384641528741888 to 7333384641433817088. Current Snapshot: MvccSnapshot[applied={T|T < 7333384641528741888}]
I20260925 22:58:29.770573 6062 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:58:29.770745 6062 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:58:29.773581 6062 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7f469704e86423d85c949911eb6f916: Sending DeleteTablet for 3 replicas of tablet 98316ec5887e4a0ca8ab7698fa4d4cf0
I20260925 22:58:29.774608 6184 tablet_service.cc:1514] Processing DeleteTablet for tablet 98316ec5887e4a0ca8ab7698fa4d4cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:58:29 UTC) from {username='slave'} at 127.0.0.1:35602
I20260925 22:58:29.774596 6446 tablet_service.cc:1514] Processing DeleteTablet for tablet 98316ec5887e4a0ca8ab7698fa4d4cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:58:29 UTC) from {username='slave'} at 127.0.0.1:55098
I20260925 22:58:29.774760 6315 tablet_service.cc:1514] Processing DeleteTablet for tablet 98316ec5887e4a0ca8ab7698fa4d4cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:58:29 UTC) from {username='slave'} at 127.0.0.1:39398
I20260925 22:58:29.775270 6552 tablet_replica.cc:330] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P ee4dd8e473dd4980bfee939bf1ccbb82: stopping tablet replica
I20260925 22:58:29.775552 6553 tablet_replica.cc:330] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c: stopping tablet replica
I20260925 22:58:29.775657 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 6122
I20260925 22:58:29.777218 6553 raft_consensus.cc:2229] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:58:29.777714 6553 raft_consensus.cc:2258] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:58:29.779726 6553 ts_tablet_manager.cc:1905] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 22:58:29.783914 6049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.114.193:44021 (error 108)
I20260925 22:58:29.784039 6554 tablet_replica.cc:330] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P cf0479134c2745adb1020f8ac5b134d8: stopping tablet replica
I20260925 22:58:29.784432 6554 raft_consensus.cc:2229] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P cf0479134c2745adb1020f8ac5b134d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:58:29.784842 6554 raft_consensus.cc:2258] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P cf0479134c2745adb1020f8ac5b134d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:58:29.785279 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 6253
I20260925 22:58:29.785332 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:29.784938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:44021 (local address 127.0.0.1:35612)
0925 22:58:29.785153 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:29.785232 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.193:44021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260925 22:58:29.785996 6049 catalog_manager.cc:4584] TS ee4dd8e473dd4980bfee939bf1ccbb82 (127.20.114.193:44021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 98316ec5887e4a0ca8ab7698fa4d4cf0: Network error: Client connection negotiation failed: client connection to 127.20.114.193:44021: connect: Connection refused (error 111)
I20260925 22:58:29.786223 6553 ts_tablet_manager.cc:1918] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:58:29.786347 6553 log.cc:1196] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/98316ec5887e4a0ca8ab7698fa4d4cf0
I20260925 22:58:29.786723 6553 ts_tablet_manager.cc:1939] T 98316ec5887e4a0ca8ab7698fa4d4cf0 P 79d63aca23e341b1bd945ea11be8669c: Deleting consensus metadata
I20260925 22:58:29.788159 6047 catalog_manager.cc:4841] TS 79d63aca23e341b1bd945ea11be8669c (127.20.114.195:42503): tablet 98316ec5887e4a0ca8ab7698fa4d4cf0 (table test-table [id=0542968351b94c4aba132045f3a3501e]) successfully deleted
I20260925 22:58:29.792232 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 6384
I20260925 22:58:29.792533 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:29.792163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:37555 (local address 127.0.0.1:39408)
0925 22:58:29.792340 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:29.792412 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.194:37555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 22:58:29.799417 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 6031
2026-09-25T22:58:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1094 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:58:29 dist-test-slave-x1lh krb5kdc[6558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:58:29 dist-test-slave-x1lh krb5kdc[6558](info): set up 2 sockets
Sep 25 22:58:29 dist-test-slave-x1lh krb5kdc[6558](info): commencing operation
krb5kdc: starting...
W20260925 22:58:30.881505 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:58:30 dist-test-slave-x1lh krb5kdc[6558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:58:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:58:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:58:31.016584 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:42629
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:42629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:31.136529 6571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:31.136945 6571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:31.137065 6571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:31.141317 6571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:58:31.141441 6571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:31.141477 6571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:31.141515 6571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:58:31.141551 6571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:58:31.147271 6571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:42629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:42629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:31.148519 6571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:31.149621 6571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:31.158457 6579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:31.158499 6576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:31.158452 6577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:31.158843 6571 server_base.cc:1034] running on GCE node
I20260925 22:58:31.159983 6571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:31.161293 6571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:31.162474 6571 hybrid_clock.cc:648] HybridClock initialized: now 1790377111162458 us; error 38 us; skew 500 ppm
Sep 25 22:58:31 dist-test-slave-x1lh krb5kdc[6558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377111, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:31.165518 6571 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:58:31.166661 6571 webserver.cc:460] Webserver started at http://127.20.114.254:39897/ using document root <none> and password file <none>
I20260925 22:58:31.167299 6571 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:31.167397 6571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:31.167647 6571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:31.169996 6571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "bb755823d2924be19bbc961b5b9750fc"
format_stamp: "Formatted at 2026-09-25 22:58:31 on dist-test-slave-x1lh"
server_key: "76d002e316abe62cd70d2f0c45e634cb"
server_key_iv: "3d0967c8c9d71abcb8da4735bcf01d14"
server_key_version: "encryptionkey@0"
I20260925 22:58:31.170614 6571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "bb755823d2924be19bbc961b5b9750fc"
format_stamp: "Formatted at 2026-09-25 22:58:31 on dist-test-slave-x1lh"
server_key: "76d002e316abe62cd70d2f0c45e634cb"
server_key_iv: "3d0967c8c9d71abcb8da4735bcf01d14"
server_key_version: "encryptionkey@0"
I20260925 22:58:31.174667 6571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:58:31.177445 6586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:31.178718 6571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:31.178886 6571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "bb755823d2924be19bbc961b5b9750fc"
format_stamp: "Formatted at 2026-09-25 22:58:31 on dist-test-slave-x1lh"
server_key: "76d002e316abe62cd70d2f0c45e634cb"
server_key_iv: "3d0967c8c9d71abcb8da4735bcf01d14"
server_key_version: "encryptionkey@0"
I20260925 22:58:31.179030 6571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:31.196450 6571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:31.199930 6571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:31.200168 6571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:31.208040 6571 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:42629
I20260925 22:58:31.208060 6638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:42629 every 8 connection(s)
I20260925 22:58:31.209137 6571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:58:31.212136 6639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:31.214088 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 6571
I20260925 22:58:31.214218 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:58:31.214507 20939 external_mini_cluster.cc:1428] Setting key 5cfa28c93c81cc06fd2705266fcc1ee1
I20260925 22:58:31.219202 6639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc: Bootstrap starting.
I20260925 22:58:31.222008 6639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:31.222998 6639 log.cc:824] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:58:31 dist-test-slave-x1lh krb5kdc[6558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:58:31.225540 6639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc: No bootstrap required, opened a new log
I20260925 22:58:31.229681 6639 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb755823d2924be19bbc961b5b9750fc" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 42629 } }
I20260925 22:58:31.229941 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:31.216096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59676 (local address 127.20.114.254:42629)
0925 22:58:31.216596 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:31.216611 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:31.216648 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:58:31.217408 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:31.217427 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:31.217766 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:31.218136 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:31.218971 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:31.220026 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:31.220979 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:31.221483 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:31.221679 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:31.224667 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:31.224673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:31.224688 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:31.224718 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:31.226771 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:31.227337 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:31.227343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:31.227347 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:31.227470 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:31.227769 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:31.227772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:31.227774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:31.228217 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:31.228455 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:31.228770 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 22:58:31.229124 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":165,"threads_started":1}
I20260925 22:58:31.229972 6639 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:31.230113 6639 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb755823d2924be19bbc961b5b9750fc, State: Initialized, Role: FOLLOWER
I20260925 22:58:31.230576 6639 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb755823d2924be19bbc961b5b9750fc" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 42629 } }
I20260925 22:58:31.230748 6639 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:58:31.230924 6639 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:58:31.231069 6639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:31.232121 6639 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb755823d2924be19bbc961b5b9750fc" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 42629 } }
I20260925 22:58:31.232545 6639 leader_election.cc:302] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb755823d2924be19bbc961b5b9750fc; no voters:
I20260925 22:58:31.233035 6639 leader_election.cc:288] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:58:31.233124 6644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:31.233361 6644 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc [term 1 LEADER]: Becoming Leader. State: Replica: bb755823d2924be19bbc961b5b9750fc, State: Running, Role: LEADER
I20260925 22:58:31.233680 6644 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb755823d2924be19bbc961b5b9750fc" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 42629 } }
I20260925 22:58:31.234366 6639 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:58:31.235621 6646 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb755823d2924be19bbc961b5b9750fc. Latest consensus state: current_term: 1 leader_uuid: "bb755823d2924be19bbc961b5b9750fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb755823d2924be19bbc961b5b9750fc" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 42629 } } }
I20260925 22:58:31.235773 6646 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:31.236039 6645 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb755823d2924be19bbc961b5b9750fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb755823d2924be19bbc961b5b9750fc" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 42629 } } }
I20260925 22:58:31.236145 6645 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:31.236215 6653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:58:31.239503 6653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:58:31.245630 6653 catalog_manager.cc:1347] Generated new cluster ID: e546bfec827949d9926187a355cc4be5
I20260925 22:58:31.245728 6653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:58:31.268213 6653 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:58:31.270054 6653 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:58:31.277204 6653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc: Generated new TSK 0
I20260925 22:58:31.278180 6653 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.114.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
I20260925 22:58:31.317761 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:42629
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:31.443427 6665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:31.443800 6665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:31.443878 6665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:31.448187 6665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:31.448311 6665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:31.448415 6665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:58:31.454167 6665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:42629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:31.455394 6665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:31.456527 6665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:31.467469 6671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:31.467469 6670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:31.467756 6665 server_base.cc:1034] running on GCE node
W20260925 22:58:31.467953 6673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:31.468469 6665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:31.469256 6665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:31.470468 6665 hybrid_clock.cc:648] HybridClock initialized: now 1790377111470451 us; error 36 us; skew 500 ppm
Sep 25 22:58:31 dist-test-slave-x1lh krb5kdc[6558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377111, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:31.473771 6665 init.cc:377] Logged in from keytab as kudu/127.20.114.193@KRBTEST.COM (short username kudu)
I20260925 22:58:31.474895 6665 webserver.cc:460] Webserver started at http://127.20.114.193:42285/ using document root <none> and password file <none>
I20260925 22:58:31.475535 6665 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:31.475636 6665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:31.475899 6665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:31.478266 6665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "c1b7fece0cb5413f8eda8e4d79b48ae5"
format_stamp: "Formatted at 2026-09-25 22:58:31 on dist-test-slave-x1lh"
server_key: "4e51b6c33a76f86ad879f4ed6a7bfaff"
server_key_iv: "098501ab294064dfabe33c476a1245ba"
server_key_version: "encryptionkey@0"
I20260925 22:58:31.478933 6665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "c1b7fece0cb5413f8eda8e4d79b48ae5"
format_stamp: "Formatted at 2026-09-25 22:58:31 on dist-test-slave-x1lh"
server_key: "4e51b6c33a76f86ad879f4ed6a7bfaff"
server_key_iv: "098501ab294064dfabe33c476a1245ba"
server_key_version: "encryptionkey@0"
I20260925 22:58:31.483677 6665 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:58:31.486830 6680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:31.488469 6665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:58:31.488667 6665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "c1b7fece0cb5413f8eda8e4d79b48ae5"
format_stamp: "Formatted at 2026-09-25 22:58:31 on dist-test-slave-x1lh"
server_key: "4e51b6c33a76f86ad879f4ed6a7bfaff"
server_key_iv: "098501ab294064dfabe33c476a1245ba"
server_key_version: "encryptionkey@0"
I20260925 22:58:31.488826 6665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:31.499995 6665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:31.503999 6665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:31.504315 6665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:31.505177 6665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:58:31.506160 6665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:58:31.506242 6665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:31.506319 6665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:58:31.506372 6665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:31.516496 6665 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:37395
I20260925 22:58:31.516521 6793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:37395 every 8 connection(s)
I20260925 22:58:31.517546 6665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:58:31.526181 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 6665
I20260925 22:58:31.526403 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:58:31.526759 20939 external_mini_cluster.cc:1428] Setting key 647b9ce9105cd240f253dec74051d0d5
Sep 25 22:58:31 dist-test-slave-x1lh krb5kdc[6558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377111, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:58:31.535722 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:31.519596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:46935 (local address 127.20.114.254:42629)
0925 22:58:31.519754 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:31.519760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:31.520916 (+ 1156us) server_negotiation.cc:407] Connection header received
0925 22:58:31.522464 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:31.522467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:31.522521 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:31.522610 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:31.524134 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:31.524708 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:31.526548 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:31.526862 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:31.526991 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:31.530491 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:31.530497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:31.530508 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:31.530549 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:31.532569 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:31.533355 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:31.533361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:31.533364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:31.533515 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:31.534062 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:31.534069 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:31.534073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:31.534413 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:31.534553 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:31.535308 (+ 755us) server_negotiation.cc:299] Negotiation successful
0925 22:58:31.535479 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:58:31.536718 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:31.520080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:42629 (local address 127.20.114.193:46935)
0925 22:58:31.520656 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:31.520707 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:31.522204 (+ 1497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:31.522795 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:31.522803 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:31.523253 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:58:31.523786 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:31.523802 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:31.524970 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:31.524974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:31.526306 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:31.526321 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:31.527087 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:31.527094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:31.527321 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:31.528043 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:58:31.528063 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:58:31.530168 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:58:31.532751 (+ 2583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:58:31.532758 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:58:31.532782 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:58:31.533207 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:58:31.533655 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:58:31.533659 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:58:31.533660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:58:31.533922 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:58:31.534572 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:58:31.534579 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:58:31.534978 (+ 399us) client_negotiation.cc:769] Sending connection context
0925 22:58:31.535202 (+ 224us) client_negotiation.cc:240] Negotiation successful
0925 22:58:31.535649 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":134,"threads_started":1}
WARNING: no policy specified for kudu/127.20.114.194@KRBTEST.COM; defaulting to no policy
I20260925 22:58:31.538317 6794 heartbeater.cc:347] Connected to a master server at 127.20.114.254:42629
I20260925 22:58:31.538568 6794 heartbeater.cc:464] Registering TS with master...
I20260925 22:58:31.539191 6794 heartbeater.cc:511] Master 127.20.114.254:42629 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.194@KRBTEST.COM" created.
I20260925 22:58:31.541107 6602 ts_manager.cc:194] Registered new tserver with Master: c1b7fece0cb5413f8eda8e4d79b48ae5 (127.20.114.193:37395)
I20260925 22:58:31.542317 6602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:46935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
I20260925 22:58:31.560073 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:42629
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:31.677701 6799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:31.678052 6799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:31.678131 6799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:31.683229 6799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:31.683333 6799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:31.683440 6799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:58:31.689592 6799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:42629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:31.690847 6799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:31.691967 6799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:31.703006 6804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:31.703025 6805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:31.703001 6807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:31.703703 6799 server_base.cc:1034] running on GCE node
I20260925 22:58:31.704186 6799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:31.705006 6799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:31.706215 6799 hybrid_clock.cc:648] HybridClock initialized: now 1790377111706176 us; error 55 us; skew 500 ppm
Sep 25 22:58:31 dist-test-slave-x1lh krb5kdc[6558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377111, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:31.709476 6799 init.cc:377] Logged in from keytab as kudu/127.20.114.194@KRBTEST.COM (short username kudu)
I20260925 22:58:31.710561 6799 webserver.cc:460] Webserver started at http://127.20.114.194:41013/ using document root <none> and password file <none>
I20260925 22:58:31.711177 6799 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:31.711244 6799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:31.711479 6799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:31.713824 6799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "59c1b209b96e47848bdc5258d4607bb9"
format_stamp: "Formatted at 2026-09-25 22:58:31 on dist-test-slave-x1lh"
server_key: "e0acfc9bd7e5c5d91a3fd310ae0cc56b"
server_key_iv: "da87da800e1745865d60c05e8ea9c301"
server_key_version: "encryptionkey@0"
I20260925 22:58:31.714442 6799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "59c1b209b96e47848bdc5258d4607bb9"
format_stamp: "Formatted at 2026-09-25 22:58:31 on dist-test-slave-x1lh"
server_key: "e0acfc9bd7e5c5d91a3fd310ae0cc56b"
server_key_iv: "da87da800e1745865d60c05e8ea9c301"
server_key_version: "encryptionkey@0"
I20260925 22:58:31.718721 6799 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:58:31.721676 6814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:31.723374 6799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:31.723553 6799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "59c1b209b96e47848bdc5258d4607bb9"
format_stamp: "Formatted at 2026-09-25 22:58:31 on dist-test-slave-x1lh"
server_key: "e0acfc9bd7e5c5d91a3fd310ae0cc56b"
server_key_iv: "da87da800e1745865d60c05e8ea9c301"
server_key_version: "encryptionkey@0"
I20260925 22:58:31.723692 6799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:31.747064 6799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:31.750399 6799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:31.750622 6799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:31.751256 6799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:58:31.752131 6799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:58:31.752177 6799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:31.752247 6799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:58:31.752287 6799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:31.762107 6799 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:35591
I20260925 22:58:31.762184 6927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:35591 every 8 connection(s)
I20260925 22:58:31.763182 6799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:58:31.767477 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 6799
I20260925 22:58:31.767588 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:58:31.767875 20939 external_mini_cluster.cc:1428] Setting key ca86d6b1fdcfeff33015f93a8426ef41
Sep 25 22:58:31 dist-test-slave-x1lh krb5kdc[6558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377111, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.114.195@KRBTEST.COM; defaulting to no policy
I20260925 22:58:31.779060 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:31.765238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:51617 (local address 127.20.114.254:42629)
0925 22:58:31.765421 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:31.765426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:31.766427 (+ 1001us) server_negotiation.cc:407] Connection header received
0925 22:58:31.767892 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:31.767896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:31.767957 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:31.768048 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:31.769614 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:31.770160 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:31.771405 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:31.771636 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:31.771752 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:31.774503 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:31.774506 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:31.774508 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:31.774536 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:31.776359 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:31.777187 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:31.777191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:31.777195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:31.777298 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:31.777730 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:31.777734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:31.777749 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:31.778001 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:31.778119 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:31.778718 (+ 599us) server_negotiation.cc:299] Negotiation successful
0925 22:58:31.778873 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 22:58:31.780099 6930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:31.765692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:42629 (local address 127.20.114.194:51617)
0925 22:58:31.766213 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:31.766264 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:31.767650 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:31.768251 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:31.768259 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:31.768816 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:58:31.769466 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:31.769482 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:31.770324 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:31.770328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:31.771258 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:31.771268 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:31.771801 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:31.771805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:31.771997 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:31.772649 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:58:31.772667 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:58:31.774280 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:58:31.776535 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:58:31.776543 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:58:31.776571 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:58:31.777058 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:58:31.777433 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:58:31.777436 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:58:31.777439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:58:31.777613 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:58:31.778118 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:58:31.778128 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:58:31.778443 (+ 315us) client_negotiation.cc:769] Sending connection context
0925 22:58:31.778662 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 22:58:31.779058 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":141,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.195@KRBTEST.COM" created.
I20260925 22:58:31.781545 6928 heartbeater.cc:347] Connected to a master server at 127.20.114.254:42629
I20260925 22:58:31.781845 6928 heartbeater.cc:464] Registering TS with master...
I20260925 22:58:31.782485 6928 heartbeater.cc:511] Master 127.20.114.254:42629 requested a full tablet report, sending...
I20260925 22:58:31.783663 6602 ts_manager.cc:194] Registered new tserver with Master: 59c1b209b96e47848bdc5258d4607bb9 (127.20.114.194:35591)
I20260925 22:58:31.784373 6602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:51617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
I20260925 22:58:31.802181 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:42629
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:31.927280 6933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:31.927591 6933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:31.927666 6933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:31.932152 6933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:31.932243 6933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:31.932308 6933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:58:31.938395 6933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:42629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:31.939643 6933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:31.940738 6933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:31.951285 6938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:31.951289 6939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:31.951289 6941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:31.951591 6933 server_base.cc:1034] running on GCE node
I20260925 22:58:31.952311 6933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:31.953116 6933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:31.954339 6933 hybrid_clock.cc:648] HybridClock initialized: now 1790377111954312 us; error 32 us; skew 500 ppm
Sep 25 22:58:31 dist-test-slave-x1lh krb5kdc[6558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377111, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:31.957500 6933 init.cc:377] Logged in from keytab as kudu/127.20.114.195@KRBTEST.COM (short username kudu)
I20260925 22:58:31.958472 6933 webserver.cc:460] Webserver started at http://127.20.114.195:35117/ using document root <none> and password file <none>
I20260925 22:58:31.959071 6933 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:31.959134 6933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:31.959357 6933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:31.961587 6933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "c217240ea69b4a068345c1d07a840c48"
format_stamp: "Formatted at 2026-09-25 22:58:31 on dist-test-slave-x1lh"
server_key: "956404b1d964d92adb16ddb772695a35"
server_key_iv: "be05cbec9c66b64525b256164a0f2dc2"
server_key_version: "encryptionkey@0"
I20260925 22:58:31.962167 6933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "c217240ea69b4a068345c1d07a840c48"
format_stamp: "Formatted at 2026-09-25 22:58:31 on dist-test-slave-x1lh"
server_key: "956404b1d964d92adb16ddb772695a35"
server_key_iv: "be05cbec9c66b64525b256164a0f2dc2"
server_key_version: "encryptionkey@0"
I20260925 22:58:31.966078 6933 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:58:31.969004 6948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:31.970628 6933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:31.970747 6933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "c217240ea69b4a068345c1d07a840c48"
format_stamp: "Formatted at 2026-09-25 22:58:31 on dist-test-slave-x1lh"
server_key: "956404b1d964d92adb16ddb772695a35"
server_key_iv: "be05cbec9c66b64525b256164a0f2dc2"
server_key_version: "encryptionkey@0"
I20260925 22:58:31.970902 6933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:31.988734 6933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:31.992267 6933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:31.992512 6933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:31.993698 6933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:58:31.994716 6933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:58:31.994772 6933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:31.994824 6933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:58:31.994841 6933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:32.005059 6933 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:39165
I20260925 22:58:32.005079 7061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:39165 every 8 connection(s)
I20260925 22:58:32.006079 6933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:58:32.010355 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 6933
I20260925 22:58:32.010480 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:58:32.010797 20939 external_mini_cluster.cc:1428] Setting key bf4e2e9bf34ef300f13cf79d5843701f
Sep 25 22:58:32 dist-test-slave-x1lh krb5kdc[6558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377111, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:58:32.022451 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:32.008114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:49143 (local address 127.20.114.254:42629)
0925 22:58:32.008261 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:32.008265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:32.009248 (+ 983us) server_negotiation.cc:407] Connection header received
0925 22:58:32.010812 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:32.010815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:32.010891 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:32.010996 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:32.012297 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.012845 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.014213 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.014448 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.014566 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:32.017866 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:32.017870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:32.017873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:32.017907 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:32.019732 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:32.020431 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:32.020435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:32.020437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:32.020525 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:32.020969 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:32.020977 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:32.020979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:32.021183 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:32.021298 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:32.022080 (+ 782us) server_negotiation.cc:299] Negotiation successful
0925 22:58:32.022265 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:58:32.023350 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:32.008474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:42629 (local address 127.20.114.195:49143)
0925 22:58:32.009081 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:32.009129 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:32.010525 (+ 1396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:32.011209 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:32.011217 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:32.011605 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:58:32.012131 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:32.012146 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.013043 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.013047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:32.014058 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:32.014070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.014626 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.014630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:32.014820 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:32.015632 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:58:32.015659 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:58:32.017566 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:58:32.019933 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:58:32.019941 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:58:32.019971 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:58:32.020305 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:58:32.020628 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:58:32.020632 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:58:32.020634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:58:32.020808 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:58:32.021319 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:58:32.021328 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:58:32.021630 (+ 302us) client_negotiation.cc:769] Sending connection context
0925 22:58:32.021966 (+ 336us) client_negotiation.cc:240] Negotiation successful
0925 22:58:32.022391 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":129,"threads_started":1}
I20260925 22:58:32.024762 7062 heartbeater.cc:347] Connected to a master server at 127.20.114.254:42629
I20260925 22:58:32.025101 7062 heartbeater.cc:464] Registering TS with master...
I20260925 22:58:32.025686 7062 heartbeater.cc:511] Master 127.20.114.254:42629 requested a full tablet report, sending...
I20260925 22:58:32.026861 6602 ts_manager.cc:194] Registered new tserver with Master: c217240ea69b4a068345c1d07a840c48 (127.20.114.195:39165)
I20260925 22:58:32.027696 6602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:49143
I20260925 22:58:32.037734 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:58:32.048288 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:32.039933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59690 (local address 127.20.114.254:42629)
0925 22:58:32.040076 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:32.040080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:32.040225 (+ 145us) server_negotiation.cc:407] Connection header received
0925 22:58:32.040305 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:32.040307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:32.040357 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:32.040443 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:32.041242 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.041807 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.042678 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.042943 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.043035 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:32.044727 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:32.044730 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:32.044732 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:32.044761 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:32.046357 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:32.046948 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:32.046951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:32.046952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:32.047031 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:32.047315 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:32.047317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:32.047319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:32.047544 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:32.047669 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:32.048041 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 22:58:32.048150 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:58:32.052229 6602 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:58:32.055540 6602 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:58:32.072324 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:32.065960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:35591 (local address 127.0.0.1:33540)
0925 22:58:32.066720 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:32.066734 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:32.066827 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:32.067596 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:32.067602 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:32.067626 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:32.067876 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:32.067888 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.069028 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.069035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:32.070581 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:32.070594 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.071694 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.071698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:32.071823 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:32.071865 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 22:58:32.072023 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 22:58:32.072166 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":27,"thread_start_us":88,"threads_started":1}
I20260925 22:58:32.072337 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:32.066540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33540 (local address 127.20.114.194:35591)
0925 22:58:32.067011 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:32.067023 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:32.067053 (+ 30us) server_negotiation.cc:407] Connection header received
0925 22:58:32.067148 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:32.067156 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:32.067403 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:32.067582 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:32.068032 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.068824 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.070755 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.071539 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.071702 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:32.071891 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:32.072056 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 22:58:32.072166 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":130,"threads_started":1}
I20260925 22:58:32.072611 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:32.066367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:39165 (local address 127.0.0.1:43232)
0925 22:58:32.067023 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:32.067036 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:32.067113 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:32.067860 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:32.067862 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:32.067885 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:32.067943 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:32.067948 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.069499 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.069501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:32.071009 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:32.071021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.072199 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.072203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:32.072289 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:32.072315 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:58:32.072360 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 22:58:32.072432 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":56,"spinlock_wait_cycles":2432,"thread_start_us":105,"threads_started":1}
I20260925 22:58:32.073364 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:32.066824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43232 (local address 127.20.114.195:39165)
0925 22:58:32.067403 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:32.067417 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:32.067440 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:58:32.067518 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:32.067525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:32.067708 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:32.068087 (+ 379us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:32.068235 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.069358 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.071164 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.072063 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.072888 (+ 825us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:32.073047 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:32.073098 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 22:58:32.073205 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":137,"threads_started":1}
I20260925 22:58:32.073777 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:32.065881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38734 (local address 127.20.114.193:37395)
0925 22:58:32.067307 (+ 1426us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:32.067314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:32.067338 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:58:32.067413 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:32.067417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:32.067603 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:32.067745 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:32.068080 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.069325 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.071760 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.072952 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.073076 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:32.073178 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:32.073330 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 22:58:32.073637 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1308,"thread_start_us":141,"threads_started":1}
I20260925 22:58:32.073896 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:32.065667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:37395 (local address 127.0.0.1:38734)
0925 22:58:32.066220 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:32.066285 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:32.066428 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:32.067778 (+ 1350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:32.067783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:32.067806 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:32.067936 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:32.067943 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.069481 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.069488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:32.071603 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:32.071612 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.073092 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.073096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:32.073184 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:32.073207 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:58:32.073290 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 22:58:32.073359 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":1,"thread_start_us":121,"threads_started":1}
I20260925 22:58:32.076584 6862 tablet_service.cc:1467] Processing CreateTablet for tablet a7d27ea9dca54ea68b8fde92ecc2ec0b (DEFAULT_TABLE table=test-table [id=d9e3edb75a184821bd9b0f2d3c15e810]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:58:32.076584 6996 tablet_service.cc:1467] Processing CreateTablet for tablet a7d27ea9dca54ea68b8fde92ecc2ec0b (DEFAULT_TABLE table=test-table [id=d9e3edb75a184821bd9b0f2d3c15e810]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:58:32.076756 6728 tablet_service.cc:1467] Processing CreateTablet for tablet a7d27ea9dca54ea68b8fde92ecc2ec0b (DEFAULT_TABLE table=test-table [id=d9e3edb75a184821bd9b0f2d3c15e810]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:58:32.077852 6862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d27ea9dca54ea68b8fde92ecc2ec0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:32.077852 6996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d27ea9dca54ea68b8fde92ecc2ec0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:32.077852 6728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d27ea9dca54ea68b8fde92ecc2ec0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:32.085389 7079 tablet_bootstrap.cc:492] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c1b7fece0cb5413f8eda8e4d79b48ae5: Bootstrap starting.
I20260925 22:58:32.086831 7081 tablet_bootstrap.cc:492] T a7d27ea9dca54ea68b8fde92ecc2ec0b P 59c1b209b96e47848bdc5258d4607bb9: Bootstrap starting.
I20260925 22:58:32.086879 7080 tablet_bootstrap.cc:492] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48: Bootstrap starting.
I20260925 22:58:32.088032 7079 tablet_bootstrap.cc:654] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c1b7fece0cb5413f8eda8e4d79b48ae5: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:32.088830 7079 log.cc:824] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c1b7fece0cb5413f8eda8e4d79b48ae5: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:32.089280 7080 tablet_bootstrap.cc:654] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:32.089280 7081 tablet_bootstrap.cc:654] T a7d27ea9dca54ea68b8fde92ecc2ec0b P 59c1b209b96e47848bdc5258d4607bb9: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:32.090754 7080 log.cc:824] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:32.090754 7081 log.cc:824] T a7d27ea9dca54ea68b8fde92ecc2ec0b P 59c1b209b96e47848bdc5258d4607bb9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:32.092041 7079 tablet_bootstrap.cc:492] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c1b7fece0cb5413f8eda8e4d79b48ae5: No bootstrap required, opened a new log
I20260925 22:58:32.092293 7079 ts_tablet_manager.cc:1397] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c1b7fece0cb5413f8eda8e4d79b48ae5: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:58:32.093007 7080 tablet_bootstrap.cc:492] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48: No bootstrap required, opened a new log
I20260925 22:58:32.093007 7081 tablet_bootstrap.cc:492] T a7d27ea9dca54ea68b8fde92ecc2ec0b P 59c1b209b96e47848bdc5258d4607bb9: No bootstrap required, opened a new log
I20260925 22:58:32.093215 7080 ts_tablet_manager.cc:1397] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:58:32.093261 7081 ts_tablet_manager.cc:1397] T a7d27ea9dca54ea68b8fde92ecc2ec0b P 59c1b209b96e47848bdc5258d4607bb9: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:58:32.096201 7079 raft_consensus.cc:348] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c1b7fece0cb5413f8eda8e4d79b48ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7fece0cb5413f8eda8e4d79b48ae5" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37395 } } peers { permanent_uuid: "59c1b209b96e47848bdc5258d4607bb9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 35591 } } peers { permanent_uuid: "c217240ea69b4a068345c1d07a840c48" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 39165 } }
I20260925 22:58:32.096588 7079 raft_consensus.cc:374] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c1b7fece0cb5413f8eda8e4d79b48ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:32.096657 7079 raft_consensus.cc:729] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c1b7fece0cb5413f8eda8e4d79b48ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b7fece0cb5413f8eda8e4d79b48ae5, State: Initialized, Role: FOLLOWER
I20260925 22:58:32.097272 7079 consensus_queue.cc:260] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c1b7fece0cb5413f8eda8e4d79b48ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7fece0cb5413f8eda8e4d79b48ae5" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37395 } } peers { permanent_uuid: "59c1b209b96e47848bdc5258d4607bb9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 35591 } } peers { permanent_uuid: "c217240ea69b4a068345c1d07a840c48" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 39165 } }
I20260925 22:58:32.097338 7080 raft_consensus.cc:348] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7fece0cb5413f8eda8e4d79b48ae5" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37395 } } peers { permanent_uuid: "59c1b209b96e47848bdc5258d4607bb9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 35591 } } peers { permanent_uuid: "c217240ea69b4a068345c1d07a840c48" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 39165 } }
I20260925 22:58:32.097620 7080 raft_consensus.cc:374] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:32.097667 7080 raft_consensus.cc:729] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c217240ea69b4a068345c1d07a840c48, State: Initialized, Role: FOLLOWER
I20260925 22:58:32.097599 7081 raft_consensus.cc:348] T a7d27ea9dca54ea68b8fde92ecc2ec0b P 59c1b209b96e47848bdc5258d4607bb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7fece0cb5413f8eda8e4d79b48ae5" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37395 } } peers { permanent_uuid: "59c1b209b96e47848bdc5258d4607bb9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 35591 } } peers { permanent_uuid: "c217240ea69b4a068345c1d07a840c48" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 39165 } }
I20260925 22:58:32.097851 7081 raft_consensus.cc:374] T a7d27ea9dca54ea68b8fde92ecc2ec0b P 59c1b209b96e47848bdc5258d4607bb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:32.097899 7081 raft_consensus.cc:729] T a7d27ea9dca54ea68b8fde92ecc2ec0b P 59c1b209b96e47848bdc5258d4607bb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59c1b209b96e47848bdc5258d4607bb9, State: Initialized, Role: FOLLOWER
I20260925 22:58:32.098340 7081 consensus_queue.cc:260] T a7d27ea9dca54ea68b8fde92ecc2ec0b P 59c1b209b96e47848bdc5258d4607bb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7fece0cb5413f8eda8e4d79b48ae5" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37395 } } peers { permanent_uuid: "59c1b209b96e47848bdc5258d4607bb9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 35591 } } peers { permanent_uuid: "c217240ea69b4a068345c1d07a840c48" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 39165 } }
I20260925 22:58:32.098341 7080 consensus_queue.cc:260] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7fece0cb5413f8eda8e4d79b48ae5" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37395 } } peers { permanent_uuid: "59c1b209b96e47848bdc5258d4607bb9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 35591 } } peers { permanent_uuid: "c217240ea69b4a068345c1d07a840c48" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 39165 } }
I20260925 22:58:32.098513 7079 ts_tablet_manager.cc:1428] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c1b7fece0cb5413f8eda8e4d79b48ae5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:58:32.098927 6794 heartbeater.cc:503] Master 127.20.114.254:42629 was elected leader, sending a full tablet report...
I20260925 22:58:32.099377 7081 ts_tablet_manager.cc:1428] T a7d27ea9dca54ea68b8fde92ecc2ec0b P 59c1b209b96e47848bdc5258d4607bb9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:58:32.099365 7080 ts_tablet_manager.cc:1428] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:58:32.099706 7062 heartbeater.cc:503] Master 127.20.114.254:42629 was elected leader, sending a full tablet report...
I20260925 22:58:32.099706 6928 heartbeater.cc:503] Master 127.20.114.254:42629 was elected leader, sending a full tablet report...
I20260925 22:58:32.118664 7086 raft_consensus.cc:482] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:58:32.118935 7086 raft_consensus.cc:504] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7fece0cb5413f8eda8e4d79b48ae5" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37395 } } peers { permanent_uuid: "59c1b209b96e47848bdc5258d4607bb9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 35591 } } peers { permanent_uuid: "c217240ea69b4a068345c1d07a840c48" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 39165 } }
I20260925 22:58:32.120308 7086 leader_election.cc:288] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1b7fece0cb5413f8eda8e4d79b48ae5 (127.20.114.193:37395), 59c1b209b96e47848bdc5258d4607bb9 (127.20.114.194:35591)
I20260925 22:58:32.125080 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:32.120563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:37395 (local address 127.20.114.195:41845)
0925 22:58:32.120848 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:32.120897 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:32.121023 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:32.121331 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:32.121334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:32.121350 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:32.121499 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:32.121507 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.122561 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.122564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:32.123641 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:32.123651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.124561 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.124565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:32.124681 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:32.124707 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:58:32.124797 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 22:58:32.124920 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260925 22:58:32.125186 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:32.120754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:41845 (local address 127.20.114.193:37395)
0925 22:58:32.120973 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:32.120978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:32.121002 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:58:32.121181 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:32.121184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:32.121226 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:32.121320 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:32.121642 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.122422 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.123799 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.124421 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.124528 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:32.124574 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:32.124923 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 22:58:32.125046 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:58:32.125516 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:32.120943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:51543 (local address 127.20.114.194:35591)
0925 22:58:32.121075 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:32.121079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:32.121331 (+ 252us) server_negotiation.cc:407] Connection header received
0925 22:58:32.121561 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:32.121567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:32.121633 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:32.121711 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:32.122074 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.122922 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.124226 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.124784 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.124943 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:32.124988 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:32.125283 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 22:58:32.125381 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:58:32.125900 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:32.120762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:35591 (local address 127.20.114.195:51543)
0925 22:58:32.121242 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:32.121258 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:32.121420 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:32.121739 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:32.121742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:32.121761 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:32.121888 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:32.121905 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.123075 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.123078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:32.124083 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:32.124094 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.124996 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.125000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:32.125101 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:32.125124 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:58:32.125670 (+ 546us) client_negotiation.cc:240] Negotiation successful
0925 22:58:32.125756 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":207,"threads_started":1}
I20260925 22:58:32.126060 6748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d27ea9dca54ea68b8fde92ecc2ec0b" candidate_uuid: "c217240ea69b4a068345c1d07a840c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b7fece0cb5413f8eda8e4d79b48ae5" is_pre_election: true
I20260925 22:58:32.126426 6748 raft_consensus.cc:2454] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c1b7fece0cb5413f8eda8e4d79b48ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c217240ea69b4a068345c1d07a840c48 in term 0.
I20260925 22:58:32.126597 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d27ea9dca54ea68b8fde92ecc2ec0b" candidate_uuid: "c217240ea69b4a068345c1d07a840c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59c1b209b96e47848bdc5258d4607bb9" is_pre_election: true
I20260925 22:58:32.126921 6882 raft_consensus.cc:2454] T a7d27ea9dca54ea68b8fde92ecc2ec0b P 59c1b209b96e47848bdc5258d4607bb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c217240ea69b4a068345c1d07a840c48 in term 0.
I20260925 22:58:32.126965 6950 leader_election.cc:302] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1b7fece0cb5413f8eda8e4d79b48ae5, c217240ea69b4a068345c1d07a840c48; no voters:
I20260925 22:58:32.127295 7086 raft_consensus.cc:2781] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:58:32.127388 7086 raft_consensus.cc:482] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:58:32.127450 7086 raft_consensus.cc:3037] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:32.128686 7086 raft_consensus.cc:504] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7fece0cb5413f8eda8e4d79b48ae5" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37395 } } peers { permanent_uuid: "59c1b209b96e47848bdc5258d4607bb9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 35591 } } peers { permanent_uuid: "c217240ea69b4a068345c1d07a840c48" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 39165 } }
I20260925 22:58:32.129213 7086 leader_election.cc:288] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48 [CANDIDATE]: Term 1 election: Requested vote from peers c1b7fece0cb5413f8eda8e4d79b48ae5 (127.20.114.193:37395), 59c1b209b96e47848bdc5258d4607bb9 (127.20.114.194:35591)
I20260925 22:58:32.129616 6748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d27ea9dca54ea68b8fde92ecc2ec0b" candidate_uuid: "c217240ea69b4a068345c1d07a840c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b7fece0cb5413f8eda8e4d79b48ae5"
I20260925 22:58:32.129702 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d27ea9dca54ea68b8fde92ecc2ec0b" candidate_uuid: "c217240ea69b4a068345c1d07a840c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59c1b209b96e47848bdc5258d4607bb9"
I20260925 22:58:32.129787 6748 raft_consensus.cc:3037] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c1b7fece0cb5413f8eda8e4d79b48ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:32.129876 6882 raft_consensus.cc:3037] T a7d27ea9dca54ea68b8fde92ecc2ec0b P 59c1b209b96e47848bdc5258d4607bb9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:32.131021 6748 raft_consensus.cc:2454] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c1b7fece0cb5413f8eda8e4d79b48ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c217240ea69b4a068345c1d07a840c48 in term 1.
I20260925 22:58:32.131017 6882 raft_consensus.cc:2454] T a7d27ea9dca54ea68b8fde92ecc2ec0b P 59c1b209b96e47848bdc5258d4607bb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c217240ea69b4a068345c1d07a840c48 in term 1.
I20260925 22:58:32.131418 6950 leader_election.cc:302] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1b7fece0cb5413f8eda8e4d79b48ae5, c217240ea69b4a068345c1d07a840c48; no voters:
I20260925 22:58:32.131678 7086 raft_consensus.cc:2781] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:32.131999 7086 raft_consensus.cc:686] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48 [term 1 LEADER]: Becoming Leader. State: Replica: c217240ea69b4a068345c1d07a840c48, State: Running, Role: LEADER
I20260925 22:58:32.132380 7086 consensus_queue.cc:237] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7fece0cb5413f8eda8e4d79b48ae5" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37395 } } peers { permanent_uuid: "59c1b209b96e47848bdc5258d4607bb9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 35591 } } peers { permanent_uuid: "c217240ea69b4a068345c1d07a840c48" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 39165 } }
I20260925 22:58:32.135950 6599 catalog_manager.cc:5495] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48 reported cstate change: term changed from 0 to 1, leader changed from <none> to c217240ea69b4a068345c1d07a840c48 (127.20.114.195). New cstate: current_term: 1 leader_uuid: "c217240ea69b4a068345c1d07a840c48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1b7fece0cb5413f8eda8e4d79b48ae5" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59c1b209b96e47848bdc5258d4607bb9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 35591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c217240ea69b4a068345c1d07a840c48" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 39165 } health_report { overall_health: HEALTHY } } }
I20260925 22:58:32.161974 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:32.157194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43244 (local address 127.20.114.195:39165)
0925 22:58:32.157374 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:32.157377 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:32.157391 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:58:32.157468 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:32.157470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:32.157525 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:32.157629 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:58:32.157978 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.158543 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.160033 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.160294 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.160455 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:32.160938 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:58:32.161414 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:58:32.161586 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:32.161725 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 22:58:32.161785 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 22:58:32.168717 6748 raft_consensus.cc:1261] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c1b7fece0cb5413f8eda8e4d79b48ae5 [term 1 FOLLOWER]: Refusing update from remote peer c217240ea69b4a068345c1d07a840c48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:58:32.168717 6882 raft_consensus.cc:1261] T a7d27ea9dca54ea68b8fde92ecc2ec0b P 59c1b209b96e47848bdc5258d4607bb9 [term 1 FOLLOWER]: Refusing update from remote peer c217240ea69b4a068345c1d07a840c48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:58:32.169632 7089 consensus_queue.cc:1035] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b7fece0cb5413f8eda8e4d79b48ae5" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 37395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:58:32.169854 7086 consensus_queue.cc:1035] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59c1b209b96e47848bdc5258d4607bb9" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:58:32.180487 7094 mvcc.cc:205] Tried to move back new op lower bound from 7333384651433930752 to 7333384651297169408. Current Snapshot: MvccSnapshot[applied={T|T < 7333384651433930752}]
I20260925 22:58:32.181478 7093 mvcc.cc:205] Tried to move back new op lower bound from 7333384651433930752 to 7333384651297169408. Current Snapshot: MvccSnapshot[applied={T|T < 7333384651433930752}]
I20260925 22:58:32.188053 6599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:58:32.188277 6599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:58:32.191023 6599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb755823d2924be19bbc961b5b9750fc: Sending DeleteTablet for 3 replicas of tablet a7d27ea9dca54ea68b8fde92ecc2ec0b
I20260925 22:58:32.192246 6996 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d27ea9dca54ea68b8fde92ecc2ec0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:58:32 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:43232
I20260925 22:58:32.192185 6862 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d27ea9dca54ea68b8fde92ecc2ec0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:58:32 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:33540
I20260925 22:58:32.192390 6728 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d27ea9dca54ea68b8fde92ecc2ec0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:58:32 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:38734
I20260925 22:58:32.192963 7102 tablet_replica.cc:330] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c1b7fece0cb5413f8eda8e4d79b48ae5: stopping tablet replica
I20260925 22:58:32.193246 7103 tablet_replica.cc:330] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48: stopping tablet replica
I20260925 22:58:32.193485 7101 tablet_replica.cc:330] T a7d27ea9dca54ea68b8fde92ecc2ec0b P 59c1b209b96e47848bdc5258d4607bb9: stopping tablet replica
I20260925 22:58:32.193780 7103 raft_consensus.cc:2229] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:58:32.193876 7101 raft_consensus.cc:2229] T a7d27ea9dca54ea68b8fde92ecc2ec0b P 59c1b209b96e47848bdc5258d4607bb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:58:32.194260 7101 raft_consensus.cc:2258] T a7d27ea9dca54ea68b8fde92ecc2ec0b P 59c1b209b96e47848bdc5258d4607bb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:58:32.194293 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 6665
I20260925 22:58:32.195281 7103 raft_consensus.cc:2258] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:58:32.198585 7101 ts_tablet_manager.cc:1905] T a7d27ea9dca54ea68b8fde92ecc2ec0b P 59c1b209b96e47848bdc5258d4607bb9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 22:58:32.202723 6588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.114.193:37395 (error 108)
I20260925 22:58:32.202786 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 6799
I20260925 22:58:32.204625 7103 ts_tablet_manager.cc:1905] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:58:32.204716 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:32.204358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:37395 (local address 127.0.0.1:38750)
0925 22:58:32.204522 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:32.204607 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.193:37395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 22:58:32.204922 6588 catalog_manager.cc:4584] TS c1b7fece0cb5413f8eda8e4d79b48ae5 (127.20.114.193:37395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a7d27ea9dca54ea68b8fde92ecc2ec0b: Network error: Client connection negotiation failed: client connection to 127.20.114.193:37395: connect: Connection refused (error 111)
I20260925 22:58:32.207621 7103 ts_tablet_manager.cc:1918] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:58:32.207749 7103 log.cc:1196] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/a7d27ea9dca54ea68b8fde92ecc2ec0b
I20260925 22:58:32.208191 7103 ts_tablet_manager.cc:1939] T a7d27ea9dca54ea68b8fde92ecc2ec0b P c217240ea69b4a068345c1d07a840c48: Deleting consensus metadata
I20260925 22:58:32.210114 6587 catalog_manager.cc:4841] TS c217240ea69b4a068345c1d07a840c48 (127.20.114.195:39165): tablet a7d27ea9dca54ea68b8fde92ecc2ec0b (table test-table [id=d9e3edb75a184821bd9b0f2d3c15e810]) successfully deleted
I20260925 22:58:32.211817 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:32.211521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:35591 (local address 127.0.0.1:33544)
0925 22:58:32.211661 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:32.211723 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.194:35591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 22:58:32.211933 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 6933
I20260925 22:58:32.218761 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 6571
2026-09-25T22:58:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2420 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T22:58:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:58:32Z Disabled control of system clock
I20260925 22:58:32.251767 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:44903
--webserver_interface=127.20.114.254
--webserver_port=0
--builtin_ntp_servers=127.20.114.212:33257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:44903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:58:32.374055 7108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:32.374360 7108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:32.374398 7108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:32.378764 7108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:58:32.378846 7108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:32.378863 7108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:32.378881 7108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:58:32.378897 7108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:58:32.384243 7108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:33257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:44903
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:44903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:32.385437 7108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:32.386847 7108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:32.397274 7116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:32.397230 7114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:32.397230 7113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:32.397296 7108 server_base.cc:1034] running on GCE node
I20260925 22:58:32.398288 7108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:32.399597 7108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:32.400789 7108 hybrid_clock.cc:648] HybridClock initialized: now 1790377112400762 us; error 51 us; skew 500 ppm
I20260925 22:58:32.403047 7108 webserver.cc:460] Webserver started at http://127.20.114.254:42635/ using document root <none> and password file <none>
I20260925 22:58:32.403666 7108 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:32.403740 7108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:32.403993 7108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:32.406492 7108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "71d124eeac7343c6812943e9c9e0e32a"
format_stamp: "Formatted at 2026-09-25 22:58:32 on dist-test-slave-x1lh"
server_key: "cff9bad19a309435bacbcb61aedeaa1f"
server_key_iv: "2c83b48d5b0bef6f29b8bff772fa9260"
server_key_version: "encryptionkey@0"
I20260925 22:58:32.407119 7108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "71d124eeac7343c6812943e9c9e0e32a"
format_stamp: "Formatted at 2026-09-25 22:58:32 on dist-test-slave-x1lh"
server_key: "cff9bad19a309435bacbcb61aedeaa1f"
server_key_iv: "2c83b48d5b0bef6f29b8bff772fa9260"
server_key_version: "encryptionkey@0"
I20260925 22:58:32.411494 7108 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:58:32.414525 7122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:32.416265 7108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:58:32.416421 7108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "71d124eeac7343c6812943e9c9e0e32a"
format_stamp: "Formatted at 2026-09-25 22:58:32 on dist-test-slave-x1lh"
server_key: "cff9bad19a309435bacbcb61aedeaa1f"
server_key_iv: "2c83b48d5b0bef6f29b8bff772fa9260"
server_key_version: "encryptionkey@0"
I20260925 22:58:32.416550 7108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:32.428700 7108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:32.429533 7108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:32.429701 7108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:32.437330 7108 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:44903
I20260925 22:58:32.437404 7174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:44903 every 8 connection(s)
I20260925 22:58:32.438364 7108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:58:32.439036 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 7108
I20260925 22:58:32.439195 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:58:32.439507 20939 external_mini_cluster.cc:1428] Setting key e5d390fbb01abe1f90e1e14b84f48035
I20260925 22:58:32.442070 7175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:32.448846 7175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a: Bootstrap starting.
I20260925 22:58:32.449361 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:32.441121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39574 (local address 127.20.114.254:44903)
0925 22:58:32.441782 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:32.441798 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:32.441834 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:58:32.442564 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:32.442583 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:32.442989 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:32.443277 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:32.443705 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.444713 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.445944 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.446425 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.446541 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:32.447445 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:32.447450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:32.447462 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:32.447854 (+ 392us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:32.447920 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:32.447944 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:32.448021 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:32.448182 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 22:58:32.448499 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":149,"threads_started":1}
I20260925 22:58:32.451606 7175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:32.452473 7175 log.cc:824] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:32.454594 7175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a: No bootstrap required, opened a new log
I20260925 22:58:32.458775 7175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d124eeac7343c6812943e9c9e0e32a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44903 } }
I20260925 22:58:32.459057 7175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:32.459124 7175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d124eeac7343c6812943e9c9e0e32a, State: Initialized, Role: FOLLOWER
I20260925 22:58:32.459596 7175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d124eeac7343c6812943e9c9e0e32a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44903 } }
I20260925 22:58:32.459729 7175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:58:32.459801 7175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:58:32.459930 7175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:32.461084 7175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d124eeac7343c6812943e9c9e0e32a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44903 } }
I20260925 22:58:32.461473 7175 leader_election.cc:302] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71d124eeac7343c6812943e9c9e0e32a; no voters:
I20260925 22:58:32.461835 7175 leader_election.cc:288] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:58:32.462160 7180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:32.462383 7180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a [term 1 LEADER]: Becoming Leader. State: Replica: 71d124eeac7343c6812943e9c9e0e32a, State: Running, Role: LEADER
I20260925 22:58:32.462746 7180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d124eeac7343c6812943e9c9e0e32a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44903 } }
I20260925 22:58:32.463245 7175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:58:32.464855 7181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71d124eeac7343c6812943e9c9e0e32a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d124eeac7343c6812943e9c9e0e32a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44903 } } }
I20260925 22:58:32.465041 7181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:32.465075 7182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71d124eeac7343c6812943e9c9e0e32a. Latest consensus state: current_term: 1 leader_uuid: "71d124eeac7343c6812943e9c9e0e32a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d124eeac7343c6812943e9c9e0e32a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 44903 } } }
I20260925 22:58:32.465193 7182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:32.465571 7193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:58:32.469254 7193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:58:32.475874 7193 catalog_manager.cc:1347] Generated new cluster ID: 7fa46fc39d614d119627b14e177d7c0b
I20260925 22:58:32.476052 7193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:58:32.506837 7193 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:58:32.507866 7193 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:58:32.522138 7193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a: Generated new TSK 0
I20260925 22:58:32.522951 7193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:58:32.529973 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44903
--builtin_ntp_servers=127.20.114.212:33257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:58:32.649331 7199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:32.649688 7199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:32.649788 7199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:32.654311 7199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:32.654421 7199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:32.654522 7199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:58:32.659699 7199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:33257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:32.661039 7199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:32.662511 7199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:32.673846 7207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:32.673846 7204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:32.673851 7205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:32.674352 7199 server_base.cc:1034] running on GCE node
I20260925 22:58:32.674862 7199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:32.675588 7199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:32.676818 7199 hybrid_clock.cc:648] HybridClock initialized: now 1790377112676773 us; error 55 us; skew 500 ppm
I20260925 22:58:32.678993 7199 webserver.cc:460] Webserver started at http://127.20.114.193:46355/ using document root <none> and password file <none>
I20260925 22:58:32.679627 7199 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:32.679724 7199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:32.679975 7199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:32.682336 7199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "762da26cfacd40e094bbb521e7abd7ca"
format_stamp: "Formatted at 2026-09-25 22:58:32 on dist-test-slave-x1lh"
server_key: "c8008eff17d58f84a0d661c8f30db5a1"
server_key_iv: "1cafefc40a9d942838d473782fc4060f"
server_key_version: "encryptionkey@0"
I20260925 22:58:32.682952 7199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "762da26cfacd40e094bbb521e7abd7ca"
format_stamp: "Formatted at 2026-09-25 22:58:32 on dist-test-slave-x1lh"
server_key: "c8008eff17d58f84a0d661c8f30db5a1"
server_key_iv: "1cafefc40a9d942838d473782fc4060f"
server_key_version: "encryptionkey@0"
I20260925 22:58:32.687143 7199 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:58:32.690371 7213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:32.691892 7199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:58:32.692075 7199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "762da26cfacd40e094bbb521e7abd7ca"
format_stamp: "Formatted at 2026-09-25 22:58:32 on dist-test-slave-x1lh"
server_key: "c8008eff17d58f84a0d661c8f30db5a1"
server_key_iv: "1cafefc40a9d942838d473782fc4060f"
server_key_version: "encryptionkey@0"
I20260925 22:58:32.692229 7199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:32.710158 7199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:32.710940 7199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:32.711158 7199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:32.711782 7199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:58:32.712788 7199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:58:32.712891 7199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:32.712991 7199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:58:32.713019 7199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:32.723235 7199 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:42097
I20260925 22:58:32.723254 7326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:42097 every 8 connection(s)
I20260925 22:58:32.724221 7199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:58:32.727607 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 7199
I20260925 22:58:32.727751 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:58:32.728140 20939 external_mini_cluster.cc:1428] Setting key e22aa4d53dffa5ae8afc4be2d9279f8b
I20260925 22:58:32.730479 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44903
--builtin_ntp_servers=127.20.114.212:33257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:58:32.736104 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:32.726334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:34573 (local address 127.20.114.254:44903)
0925 22:58:32.726490 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:32.726495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:32.728028 (+ 1533us) server_negotiation.cc:407] Connection header received
0925 22:58:32.729867 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:32.729869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:32.729913 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:32.729972 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:32.731434 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.732182 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.733716 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.733969 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.734106 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:32.735495 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:32.735499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:32.735504 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:32.735525 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:32.735544 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:32.735553 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:32.735642 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:32.735821 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 22:58:32.735930 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:58:32.737139 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:32.726878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:44903 (local address 127.20.114.193:34573)
0925 22:58:32.727796 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:32.727867 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:32.729587 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:32.730191 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:32.730205 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:32.730345 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:58:32.731227 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:32.731254 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.732398 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.732401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:32.733557 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:32.733569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.734170 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.734177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:32.734437 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:32.735172 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:58:32.735174 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:58:32.735224 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:58:32.735303 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:58:32.735645 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:58:32.735659 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:58:32.735667 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 22:58:32.735823 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 22:58:32.736160 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":128,"threads_started":1}
I20260925 22:58:32.738723 7327 heartbeater.cc:347] Connected to a master server at 127.20.114.254:44903
I20260925 22:58:32.738996 7327 heartbeater.cc:464] Registering TS with master...
I20260925 22:58:32.739586 7327 heartbeater.cc:511] Master 127.20.114.254:44903 requested a full tablet report, sending...
I20260925 22:58:32.741580 7139 ts_manager.cc:194] Registered new tserver with Master: 762da26cfacd40e094bbb521e7abd7ca (127.20.114.193:42097)
I20260925 22:58:32.743371 7139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.114.193:34573
W20260925 22:58:32.852257 7330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:32.852576 7330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:32.852617 7330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:32.857036 7330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:32.857115 7330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:32.857177 7330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:58:32.862293 7330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:33257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:32.863557 7330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:32.865536 7330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:32.876623 7336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:32.876606 7335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:32.876606 7338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:32.876842 7330 server_base.cc:1034] running on GCE node
I20260925 22:58:32.877467 7330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:32.878193 7330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:32.879417 7330 hybrid_clock.cc:648] HybridClock initialized: now 1790377112879374 us; error 57 us; skew 500 ppm
I20260925 22:58:32.881603 7330 webserver.cc:460] Webserver started at http://127.20.114.194:40511/ using document root <none> and password file <none>
I20260925 22:58:32.882279 7330 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:32.882354 7330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:32.882584 7330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:32.884793 7330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "604269669b634f0fb28ab5481260421c"
format_stamp: "Formatted at 2026-09-25 22:58:32 on dist-test-slave-x1lh"
server_key: "697b1d14c4876eb20ea008fa6a0ab869"
server_key_iv: "52b0c918423a9f00b7535d60f1a3c583"
server_key_version: "encryptionkey@0"
I20260925 22:58:32.885465 7330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "604269669b634f0fb28ab5481260421c"
format_stamp: "Formatted at 2026-09-25 22:58:32 on dist-test-slave-x1lh"
server_key: "697b1d14c4876eb20ea008fa6a0ab869"
server_key_iv: "52b0c918423a9f00b7535d60f1a3c583"
server_key_version: "encryptionkey@0"
I20260925 22:58:32.889564 7330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:58:32.892356 7344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:32.893729 7330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:32.893926 7330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "604269669b634f0fb28ab5481260421c"
format_stamp: "Formatted at 2026-09-25 22:58:32 on dist-test-slave-x1lh"
server_key: "697b1d14c4876eb20ea008fa6a0ab869"
server_key_iv: "52b0c918423a9f00b7535d60f1a3c583"
server_key_version: "encryptionkey@0"
I20260925 22:58:32.894063 7330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:32.905855 7330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:32.906636 7330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:32.906854 7330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:32.907500 7330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:58:32.908511 7330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:58:32.908582 7330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:32.908658 7330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:58:32.908708 7330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:32.918705 7330 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:35023
I20260925 22:58:32.918737 7457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:35023 every 8 connection(s)
I20260925 22:58:32.919674 7330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:58:32.928131 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 7330
I20260925 22:58:32.928300 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:58:32.928654 20939 external_mini_cluster.cc:1428] Setting key 4351373eeead4498248a22d040209243
I20260925 22:58:32.929922 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:32.921870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:44673 (local address 127.20.114.254:44903)
0925 22:58:32.922019 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:32.922023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:32.923237 (+ 1214us) server_negotiation.cc:407] Connection header received
0925 22:58:32.924631 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:32.924634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:32.924683 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:32.924761 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:32.925729 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.926260 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.927445 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.927663 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.927747 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:32.929334 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:32.929339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:32.929343 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:32.929361 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:32.929393 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:32.929404 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:32.929468 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:32.929706 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 22:58:32.929783 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:58:32.930915 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:32.922304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:44903 (local address 127.20.114.194:44673)
0925 22:58:32.923070 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:32.923118 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:32.924434 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:32.924991 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:32.924998 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:32.925086 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:58:32.925588 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:32.925601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.926384 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.926387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:32.927312 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:32.927322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:32.927949 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:32.927957 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:32.928182 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:32.928990 (+ 808us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:58:32.928992 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:58:32.929051 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:58:32.929154 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:58:32.929518 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:58:32.929532 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:58:32.929539 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 22:58:32.929696 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 22:58:32.930012 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":120,"threads_started":1}
I20260925 22:58:32.930963 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44903
--builtin_ntp_servers=127.20.114.212:33257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:58:32.932446 7458 heartbeater.cc:347] Connected to a master server at 127.20.114.254:44903
I20260925 22:58:32.932768 7458 heartbeater.cc:464] Registering TS with master...
I20260925 22:58:32.933511 7458 heartbeater.cc:511] Master 127.20.114.254:44903 requested a full tablet report, sending...
I20260925 22:58:32.934880 7139 ts_manager.cc:194] Registered new tserver with Master: 604269669b634f0fb28ab5481260421c (127.20.114.194:35023)
I20260925 22:58:32.935662 7139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.114.194:44673
W20260925 22:58:33.051604 7461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:33.051959 7461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:33.052034 7461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:33.056249 7461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:33.056370 7461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:33.056475 7461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:58:33.062017 7461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:33257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:44903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:33.063351 7461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:33.064965 7461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:58:33.075567 7461 server_base.cc:1034] running on GCE node
W20260925 22:58:33.075474 7466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:33.075522 7467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:33.075508 7469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:33.076256 7461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:33.077073 7461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:33.078269 7461 hybrid_clock.cc:648] HybridClock initialized: now 1790377113078251 us; error 31 us; skew 500 ppm
I20260925 22:58:33.080319 7461 webserver.cc:460] Webserver started at http://127.20.114.195:32893/ using document root <none> and password file <none>
I20260925 22:58:33.081105 7461 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:33.081204 7461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:33.081434 7461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:33.083660 7461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "fa69d4a7ce5f456f9bf4adc603588f8c"
format_stamp: "Formatted at 2026-09-25 22:58:33 on dist-test-slave-x1lh"
server_key: "f955049717a53fc29782d89f6e6ef052"
server_key_iv: "22ac84219622cfed1156523848f931aa"
server_key_version: "encryptionkey@0"
I20260925 22:58:33.084301 7461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "fa69d4a7ce5f456f9bf4adc603588f8c"
format_stamp: "Formatted at 2026-09-25 22:58:33 on dist-test-slave-x1lh"
server_key: "f955049717a53fc29782d89f6e6ef052"
server_key_iv: "22ac84219622cfed1156523848f931aa"
server_key_version: "encryptionkey@0"
I20260925 22:58:33.088333 7461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:58:33.091262 7475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:33.092667 7461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 22:58:33.092971 7461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "fa69d4a7ce5f456f9bf4adc603588f8c"
format_stamp: "Formatted at 2026-09-25 22:58:33 on dist-test-slave-x1lh"
server_key: "f955049717a53fc29782d89f6e6ef052"
server_key_iv: "22ac84219622cfed1156523848f931aa"
server_key_version: "encryptionkey@0"
I20260925 22:58:33.093116 7461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:33.107574 7461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:33.108361 7461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:33.108578 7461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:33.109292 7461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:58:33.110316 7461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:58:33.110387 7461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:33.110451 7461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:58:33.110503 7461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:33.119895 7461 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:44371
I20260925 22:58:33.119966 7588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:44371 every 8 connection(s)
I20260925 22:58:33.121004 7461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:58:33.129189 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 7461
I20260925 22:58:33.129374 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:58:33.129659 20939 external_mini_cluster.cc:1428] Setting key d37f2ebd3d8f15e8bda8f2b54444da78
I20260925 22:58:33.132150 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:33.123220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:34683 (local address 127.20.114.254:44903)
0925 22:58:33.123424 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:33.123428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:33.124615 (+ 1187us) server_negotiation.cc:407] Connection header received
0925 22:58:33.126227 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:33.126230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:33.126287 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:33.126372 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:33.127386 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.127997 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.129442 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.129679 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.129827 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:33.131492 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:33.131495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:33.131497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:33.131510 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:33.131531 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:33.131540 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:33.131613 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:33.131916 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 22:58:33.131993 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260925 22:58:33.133104 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:33.123680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:44903 (local address 127.20.114.195:34683)
0925 22:58:33.124442 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:33.124489 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:33.125978 (+ 1489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:33.126596 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:33.126604 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:33.126699 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:58:33.127233 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:33.127247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.128146 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.128149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:33.129292 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:33.129304 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.129876 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.129880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:33.130127 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:33.131080 (+ 953us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:58:33.131084 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:58:33.131151 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:58:33.131297 (+ 146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:58:33.131636 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:58:33.131645 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:58:33.131651 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:58:33.131789 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 22:58:33.132109 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":134,"threads_started":1}
I20260925 22:58:33.134346 7589 heartbeater.cc:347] Connected to a master server at 127.20.114.254:44903
I20260925 22:58:33.134616 7589 heartbeater.cc:464] Registering TS with master...
I20260925 22:58:33.135213 7589 heartbeater.cc:511] Master 127.20.114.254:44903 requested a full tablet report, sending...
I20260925 22:58:33.136420 7139 ts_manager.cc:194] Registered new tserver with Master: fa69d4a7ce5f456f9bf4adc603588f8c (127.20.114.195:44371)
I20260925 22:58:33.137065 7139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.114.195:34683
I20260925 22:58:33.144457 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:58:33.150333 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:33.146701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39586 (local address 127.20.114.254:44903)
0925 22:58:33.146833 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:33.146837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:33.146930 (+ 93us) server_negotiation.cc:407] Connection header received
0925 22:58:33.147042 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:33.147044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:33.147087 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:33.147157 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:33.147420 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.147920 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.148730 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.148972 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.149051 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:33.149921 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:33.149924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:33.149926 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:33.149937 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:33.149954 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:33.149962 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:33.150059 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:33.150185 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 22:58:33.150219 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:58:33.153597 7139 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:58:33.156971 7139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:58:33.172408 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:33.166376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:35023 (local address 127.0.0.1:59698)
0925 22:58:33.167437 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:33.167450 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:33.167535 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:33.168141 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:33.168148 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:33.168180 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:33.168363 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:33.168370 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.169761 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.169765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:33.171002 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:33.171012 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.171944 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.171947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:33.172025 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:33.172056 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 22:58:33.172136 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 22:58:33.172228 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"mutex_wait_us":80,"spinlock_wait_cycles":2944,"thread_start_us":118,"threads_started":1}
I20260925 22:58:33.172472 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:33.166606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59698 (local address 127.20.114.194:35023)
0925 22:58:33.167612 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:33.167629 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:33.167661 (+ 32us) server_negotiation.cc:407] Connection header received
0925 22:58:33.167742 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:33.167748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:33.167976 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:33.168289 (+ 313us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:33.168453 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.169606 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.171121 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.171807 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.171932 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:33.172117 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:33.172171 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 22:58:33.172275 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":815,"thread_start_us":102,"threads_started":1}
I20260925 22:58:33.173174 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:33.167053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:44371 (local address 127.0.0.1:50988)
0925 22:58:33.167937 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:33.167950 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:33.168048 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:33.168603 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:33.168605 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:33.168621 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:33.168696 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:33.168705 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.170050 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.170053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:33.171357 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:33.171365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.172390 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.172393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:33.172846 (+ 453us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:33.172873 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:58:33.172970 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 22:58:33.173045 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":42,"thread_start_us":83,"threads_started":1}
I20260925 22:58:33.173367 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:33.167445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50988 (local address 127.20.114.195:44371)
0925 22:58:33.167795 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:33.167806 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:33.168050 (+ 244us) server_negotiation.cc:407] Connection header received
0925 22:58:33.168153 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:33.168158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:33.168319 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:33.168464 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:33.168830 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.169957 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.171488 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.172253 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.172379 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:33.172546 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:33.173004 (+ 458us) server_negotiation.cc:299] Negotiation successful
0925 22:58:33.173108 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":130,"threads_started":1}
I20260925 22:58:33.173979 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:33.166282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:42097 (local address 127.0.0.1:39858)
0925 22:58:33.167014 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:33.167067 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:33.167207 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:33.168141 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:33.168148 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:33.168186 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:33.168372 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:33.168382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.169649 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.169653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:33.171354 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:33.171365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.173579 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.173583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:33.173690 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:33.173711 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:58:33.173775 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 22:58:33.173848 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":325,"spinlock_wait_cycles":4864,"thread_start_us":114,"threads_started":1}
I20260925 22:58:33.174079 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:33.167027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39858 (local address 127.20.114.193:42097)
0925 22:58:33.167680 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:33.167692 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:33.167718 (+ 26us) server_negotiation.cc:407] Connection header received
0925 22:58:33.167799 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:33.167802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:33.167948 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:33.168529 (+ 581us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:33.168668 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.169502 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.172619 (+ 3117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.173470 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.173591 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:33.173765 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:33.173818 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 22:58:33.173921 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":131,"threads_started":1}
I20260925 22:58:33.176397 7392 tablet_service.cc:1467] Processing CreateTablet for tablet a3ba7b1cae3c4f15ace7d3963d9ffa78 (DEFAULT_TABLE table=test-table [id=44571d72cb26402bbab8965909783c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:58:33.176406 7523 tablet_service.cc:1467] Processing CreateTablet for tablet a3ba7b1cae3c4f15ace7d3963d9ffa78 (DEFAULT_TABLE table=test-table [id=44571d72cb26402bbab8965909783c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:58:33.177492 7523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ba7b1cae3c4f15ace7d3963d9ffa78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:33.177526 7392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ba7b1cae3c4f15ace7d3963d9ffa78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:33.179839 7261 tablet_service.cc:1467] Processing CreateTablet for tablet a3ba7b1cae3c4f15ace7d3963d9ffa78 (DEFAULT_TABLE table=test-table [id=44571d72cb26402bbab8965909783c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:58:33.181033 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ba7b1cae3c4f15ace7d3963d9ffa78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:33.184376 7606 tablet_bootstrap.cc:492] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c: Bootstrap starting.
I20260925 22:58:33.186561 7607 tablet_bootstrap.cc:492] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 604269669b634f0fb28ab5481260421c: Bootstrap starting.
I20260925 22:58:33.186837 7606 tablet_bootstrap.cc:654] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:33.187623 7606 log.cc:824] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:33.189049 7607 tablet_bootstrap.cc:654] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 604269669b634f0fb28ab5481260421c: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:33.190032 7607 log.cc:824] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 604269669b634f0fb28ab5481260421c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:33.190626 7609 tablet_bootstrap.cc:492] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 762da26cfacd40e094bbb521e7abd7ca: Bootstrap starting.
I20260925 22:58:33.190737 7606 tablet_bootstrap.cc:492] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c: No bootstrap required, opened a new log
I20260925 22:58:33.190981 7606 ts_tablet_manager.cc:1397] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:58:33.192341 7607 tablet_bootstrap.cc:492] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 604269669b634f0fb28ab5481260421c: No bootstrap required, opened a new log
I20260925 22:58:33.192551 7607 ts_tablet_manager.cc:1397] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 604269669b634f0fb28ab5481260421c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 22:58:33.193135 7609 tablet_bootstrap.cc:654] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 762da26cfacd40e094bbb521e7abd7ca: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:33.194047 7609 log.cc:824] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 762da26cfacd40e094bbb521e7abd7ca: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:33.194926 7606 raft_consensus.cc:348] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762da26cfacd40e094bbb521e7abd7ca" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 42097 } } peers { permanent_uuid: "604269669b634f0fb28ab5481260421c" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 35023 } } peers { permanent_uuid: "fa69d4a7ce5f456f9bf4adc603588f8c" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 44371 } }
I20260925 22:58:33.195182 7606 raft_consensus.cc:374] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:33.195242 7606 raft_consensus.cc:729] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa69d4a7ce5f456f9bf4adc603588f8c, State: Initialized, Role: FOLLOWER
I20260925 22:58:33.195701 7606 consensus_queue.cc:260] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762da26cfacd40e094bbb521e7abd7ca" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 42097 } } peers { permanent_uuid: "604269669b634f0fb28ab5481260421c" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 35023 } } peers { permanent_uuid: "fa69d4a7ce5f456f9bf4adc603588f8c" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 44371 } }
I20260925 22:58:33.196324 7609 tablet_bootstrap.cc:492] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 762da26cfacd40e094bbb521e7abd7ca: No bootstrap required, opened a new log
I20260925 22:58:33.196543 7609 ts_tablet_manager.cc:1397] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 762da26cfacd40e094bbb521e7abd7ca: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:58:33.196889 7589 heartbeater.cc:503] Master 127.20.114.254:44903 was elected leader, sending a full tablet report...
I20260925 22:58:33.197093 7607 raft_consensus.cc:348] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 604269669b634f0fb28ab5481260421c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762da26cfacd40e094bbb521e7abd7ca" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 42097 } } peers { permanent_uuid: "604269669b634f0fb28ab5481260421c" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 35023 } } peers { permanent_uuid: "fa69d4a7ce5f456f9bf4adc603588f8c" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 44371 } }
I20260925 22:58:33.197530 7607 raft_consensus.cc:374] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 604269669b634f0fb28ab5481260421c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:33.197587 7607 raft_consensus.cc:729] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 604269669b634f0fb28ab5481260421c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604269669b634f0fb28ab5481260421c, State: Initialized, Role: FOLLOWER
I20260925 22:58:33.198074 7607 consensus_queue.cc:260] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 604269669b634f0fb28ab5481260421c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762da26cfacd40e094bbb521e7abd7ca" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 42097 } } peers { permanent_uuid: "604269669b634f0fb28ab5481260421c" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 35023 } } peers { permanent_uuid: "fa69d4a7ce5f456f9bf4adc603588f8c" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 44371 } }
I20260925 22:58:33.198477 7606 ts_tablet_manager.cc:1428] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:58:33.199097 7607 ts_tablet_manager.cc:1428] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 604269669b634f0fb28ab5481260421c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:58:33.199456 7458 heartbeater.cc:503] Master 127.20.114.254:44903 was elected leader, sending a full tablet report...
I20260925 22:58:33.201267 7609 raft_consensus.cc:348] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 762da26cfacd40e094bbb521e7abd7ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762da26cfacd40e094bbb521e7abd7ca" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 42097 } } peers { permanent_uuid: "604269669b634f0fb28ab5481260421c" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 35023 } } peers { permanent_uuid: "fa69d4a7ce5f456f9bf4adc603588f8c" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 44371 } }
I20260925 22:58:33.201586 7609 raft_consensus.cc:374] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 762da26cfacd40e094bbb521e7abd7ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:33.201648 7609 raft_consensus.cc:729] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 762da26cfacd40e094bbb521e7abd7ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 762da26cfacd40e094bbb521e7abd7ca, State: Initialized, Role: FOLLOWER
I20260925 22:58:33.202210 7609 consensus_queue.cc:260] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 762da26cfacd40e094bbb521e7abd7ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762da26cfacd40e094bbb521e7abd7ca" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 42097 } } peers { permanent_uuid: "604269669b634f0fb28ab5481260421c" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 35023 } } peers { permanent_uuid: "fa69d4a7ce5f456f9bf4adc603588f8c" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 44371 } }
I20260925 22:58:33.202865 7327 heartbeater.cc:503] Master 127.20.114.254:44903 was elected leader, sending a full tablet report...
I20260925 22:58:33.203174 7609 ts_tablet_manager.cc:1428] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 762da26cfacd40e094bbb521e7abd7ca: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260925 22:58:33.225582 7328 tablet.cc:2369] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 762da26cfacd40e094bbb521e7abd7ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:58:33.300794 7612 raft_consensus.cc:482] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:58:33.301041 7612 raft_consensus.cc:504] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762da26cfacd40e094bbb521e7abd7ca" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 42097 } } peers { permanent_uuid: "604269669b634f0fb28ab5481260421c" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 35023 } } peers { permanent_uuid: "fa69d4a7ce5f456f9bf4adc603588f8c" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 44371 } }
I20260925 22:58:33.302191 7612 leader_election.cc:288] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 762da26cfacd40e094bbb521e7abd7ca (127.20.114.193:42097), 604269669b634f0fb28ab5481260421c (127.20.114.194:35023)
I20260925 22:58:33.306692 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:33.302645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:57745 (local address 127.20.114.193:42097)
0925 22:58:33.302782 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:33.302786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:33.302834 (+ 48us) server_negotiation.cc:407] Connection header received
0925 22:58:33.302922 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:33.302925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:33.302969 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:33.303031 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:33.303401 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.304119 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.305541 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.306071 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.306175 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:33.306215 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:33.306472 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 22:58:33.306535 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:58:33.306690 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:33.302508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:42097 (local address 127.20.114.195:57745)
0925 22:58:33.302639 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:33.302657 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:33.302762 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:33.303085 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:33.303089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:33.303109 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:33.303236 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:33.303242 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.304262 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.304266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:33.305387 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:33.305398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.306213 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.306217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:33.306329 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:33.306355 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:58:33.306445 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 22:58:33.306531 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 22:58:33.307224 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:33.302645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:46029 (local address 127.20.114.194:35023)
0925 22:58:33.302772 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:33.302777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:33.303447 (+ 670us) server_negotiation.cc:407] Connection header received
0925 22:58:33.303509 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:33.303511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:33.303559 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:33.303639 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:33.303975 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.304688 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.305920 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.306509 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.306622 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:33.306663 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:33.306985 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 22:58:33.307100 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:58:33.307597 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:33.302508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:35023 (local address 127.20.114.195:46029)
0925 22:58:33.303269 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:33.303321 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:33.303430 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:33.303677 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:33.303680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:33.303697 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:33.303820 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:33.303825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.304831 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.304833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:33.305753 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:33.305764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.306645 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.306649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:33.306754 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:33.306779 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:58:33.307400 (+ 621us) client_negotiation.cc:240] Negotiation successful
0925 22:58:33.307485 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":323,"thread_start_us":172,"threads_started":1}
I20260925 22:58:33.307580 7281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ba7b1cae3c4f15ace7d3963d9ffa78" candidate_uuid: "fa69d4a7ce5f456f9bf4adc603588f8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762da26cfacd40e094bbb521e7abd7ca" is_pre_election: true
I20260925 22:58:33.307978 7281 raft_consensus.cc:2454] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 762da26cfacd40e094bbb521e7abd7ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa69d4a7ce5f456f9bf4adc603588f8c in term 0.
I20260925 22:58:33.308267 7412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ba7b1cae3c4f15ace7d3963d9ffa78" candidate_uuid: "fa69d4a7ce5f456f9bf4adc603588f8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604269669b634f0fb28ab5481260421c" is_pre_election: true
I20260925 22:58:33.308600 7412 raft_consensus.cc:2454] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 604269669b634f0fb28ab5481260421c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa69d4a7ce5f456f9bf4adc603588f8c in term 0.
I20260925 22:58:33.308568 7476 leader_election.cc:302] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 762da26cfacd40e094bbb521e7abd7ca, fa69d4a7ce5f456f9bf4adc603588f8c; no voters:
I20260925 22:58:33.308928 7612 raft_consensus.cc:2781] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:58:33.309052 7612 raft_consensus.cc:482] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:58:33.309131 7612 raft_consensus.cc:3037] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:33.310230 7612 raft_consensus.cc:504] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762da26cfacd40e094bbb521e7abd7ca" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 42097 } } peers { permanent_uuid: "604269669b634f0fb28ab5481260421c" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 35023 } } peers { permanent_uuid: "fa69d4a7ce5f456f9bf4adc603588f8c" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 44371 } }
I20260925 22:58:33.310675 7612 leader_election.cc:288] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c [CANDIDATE]: Term 1 election: Requested vote from peers 762da26cfacd40e094bbb521e7abd7ca (127.20.114.193:42097), 604269669b634f0fb28ab5481260421c (127.20.114.194:35023)
I20260925 22:58:33.311209 7281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ba7b1cae3c4f15ace7d3963d9ffa78" candidate_uuid: "fa69d4a7ce5f456f9bf4adc603588f8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762da26cfacd40e094bbb521e7abd7ca"
I20260925 22:58:33.311208 7412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ba7b1cae3c4f15ace7d3963d9ffa78" candidate_uuid: "fa69d4a7ce5f456f9bf4adc603588f8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604269669b634f0fb28ab5481260421c"
I20260925 22:58:33.311383 7412 raft_consensus.cc:3037] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 604269669b634f0fb28ab5481260421c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:33.311367 7281 raft_consensus.cc:3037] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 762da26cfacd40e094bbb521e7abd7ca [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:33.312835 7412 raft_consensus.cc:2454] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 604269669b634f0fb28ab5481260421c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa69d4a7ce5f456f9bf4adc603588f8c in term 1.
I20260925 22:58:33.312836 7281 raft_consensus.cc:2454] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 762da26cfacd40e094bbb521e7abd7ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa69d4a7ce5f456f9bf4adc603588f8c in term 1.
I20260925 22:58:33.313277 7476 leader_election.cc:302] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 762da26cfacd40e094bbb521e7abd7ca, fa69d4a7ce5f456f9bf4adc603588f8c; no voters:
I20260925 22:58:33.313519 7612 raft_consensus.cc:2781] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:33.313747 7612 raft_consensus.cc:686] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c [term 1 LEADER]: Becoming Leader. State: Replica: fa69d4a7ce5f456f9bf4adc603588f8c, State: Running, Role: LEADER
I20260925 22:58:33.314121 7612 consensus_queue.cc:237] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762da26cfacd40e094bbb521e7abd7ca" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 42097 } } peers { permanent_uuid: "604269669b634f0fb28ab5481260421c" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 35023 } } peers { permanent_uuid: "fa69d4a7ce5f456f9bf4adc603588f8c" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 44371 } }
I20260925 22:58:33.317800 7138 catalog_manager.cc:5495] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c reported cstate change: term changed from 0 to 1, leader changed from <none> to fa69d4a7ce5f456f9bf4adc603588f8c (127.20.114.195). New cstate: current_term: 1 leader_uuid: "fa69d4a7ce5f456f9bf4adc603588f8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762da26cfacd40e094bbb521e7abd7ca" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "604269669b634f0fb28ab5481260421c" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 35023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa69d4a7ce5f456f9bf4adc603588f8c" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 44371 } health_report { overall_health: HEALTHY } } }
W20260925 22:58:33.372188 7590 tablet.cc:2369] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:58:33.373402 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:33.368553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50998 (local address 127.20.114.195:44371)
0925 22:58:33.368731 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:33.368736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:33.368752 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:58:33.368898 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:33.368903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:33.368982 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:33.369166 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:58:33.369489 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.370458 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.371444 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:33.371679 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:33.371796 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:33.372145 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:58:33.372666 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:58:33.372849 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:33.373156 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 22:58:33.373207 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 22:58:33.381556 7281 raft_consensus.cc:1261] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 762da26cfacd40e094bbb521e7abd7ca [term 1 FOLLOWER]: Refusing update from remote peer fa69d4a7ce5f456f9bf4adc603588f8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:58:33.381556 7412 raft_consensus.cc:1261] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 604269669b634f0fb28ab5481260421c [term 1 FOLLOWER]: Refusing update from remote peer fa69d4a7ce5f456f9bf4adc603588f8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:58:33.382392 7616 consensus_queue.cc:1035] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "762da26cfacd40e094bbb521e7abd7ca" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:58:33.382638 7612 consensus_queue.cc:1035] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "604269669b634f0fb28ab5481260421c" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 35023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:58:33.392010 7621 mvcc.cc:205] Tried to move back new op lower bound from 7333384656401309696 to 7333384656137347072. Current Snapshot: MvccSnapshot[applied={T|T < 7333384656401309696}]
I20260925 22:58:33.393777 7620 mvcc.cc:205] Tried to move back new op lower bound from 7333384656401309696 to 7333384656137347072. Current Snapshot: MvccSnapshot[applied={T|T < 7333384656401309696 or (T in {7333384656401309696})}]
I20260925 22:58:33.400273 7138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:58:33.400484 7138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:58:33.403690 7138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71d124eeac7343c6812943e9c9e0e32a: Sending DeleteTablet for 3 replicas of tablet a3ba7b1cae3c4f15ace7d3963d9ffa78
I20260925 22:58:33.404682 7261 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ba7b1cae3c4f15ace7d3963d9ffa78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:58:33 UTC) from {username='slave'} at 127.0.0.1:39858
I20260925 22:58:33.404661 7392 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ba7b1cae3c4f15ace7d3963d9ffa78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:58:33 UTC) from {username='slave'} at 127.0.0.1:59698
I20260925 22:58:33.405323 7628 tablet_replica.cc:330] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 604269669b634f0fb28ab5481260421c: stopping tablet replica
I20260925 22:58:33.405316 7523 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ba7b1cae3c4f15ace7d3963d9ffa78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:58:33 UTC) from {username='slave'} at 127.0.0.1:50988
I20260925 22:58:33.405714 7628 raft_consensus.cc:2229] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 604269669b634f0fb28ab5481260421c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:58:33.405887 7630 tablet_replica.cc:330] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c: stopping tablet replica
I20260925 22:58:33.406112 7628 raft_consensus.cc:2258] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 604269669b634f0fb28ab5481260421c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:58:33.406306 7630 raft_consensus.cc:2229] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:58:33.406622 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 7199
I20260925 22:58:33.408512 7630 raft_consensus.cc:2258] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:58:33.409124 7628 ts_tablet_manager.cc:1905] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 604269669b634f0fb28ab5481260421c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:58:33.410367 7630 ts_tablet_manager.cc:1905] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:58:33.412300 7628 ts_tablet_manager.cc:1918] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 604269669b634f0fb28ab5481260421c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:58:33.412436 7628 log.cc:1196] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 604269669b634f0fb28ab5481260421c: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/a3ba7b1cae3c4f15ace7d3963d9ffa78
I20260925 22:58:33.412801 7628 ts_tablet_manager.cc:1939] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P 604269669b634f0fb28ab5481260421c: Deleting consensus metadata
I20260925 22:58:33.413797 7630 ts_tablet_manager.cc:1918] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:58:33.413992 7630 log.cc:1196] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/a3ba7b1cae3c4f15ace7d3963d9ffa78
I20260925 22:58:33.414426 7630 ts_tablet_manager.cc:1939] T a3ba7b1cae3c4f15ace7d3963d9ffa78 P fa69d4a7ce5f456f9bf4adc603588f8c: Deleting consensus metadata
I20260925 22:58:33.414425 7124 catalog_manager.cc:4841] TS 604269669b634f0fb28ab5481260421c (127.20.114.194:35023): tablet a3ba7b1cae3c4f15ace7d3963d9ffa78 (table test-table [id=44571d72cb26402bbab8965909783c09]) successfully deleted
I20260925 22:58:33.415635 7123 catalog_manager.cc:4841] TS fa69d4a7ce5f456f9bf4adc603588f8c (127.20.114.195:44371): tablet a3ba7b1cae3c4f15ace7d3963d9ffa78 (table test-table [id=44571d72cb26402bbab8965909783c09]) successfully deleted
W20260925 22:58:33.415956 7123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.114.193:42097 (error 108)
I20260925 22:58:33.416219 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 7330
I20260925 22:58:33.418159 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:33.416678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:42097 (local address 127.0.0.1:39872)
0925 22:58:33.417967 (+ 1289us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:33.418056 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.193:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1182}
W20260925 22:58:33.418423 7123 catalog_manager.cc:4584] TS 762da26cfacd40e094bbb521e7abd7ca (127.20.114.193:42097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a3ba7b1cae3c4f15ace7d3963d9ffa78: Network error: Client connection negotiation failed: client connection to 127.20.114.193:42097: connect: Connection refused (error 111)
I20260925 22:58:33.426241 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 7461
I20260925 22:58:33.433825 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 7108
2026-09-25T22:58:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1213 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:58:33 dist-test-slave-x1lh krb5kdc[7634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:58:33 dist-test-slave-x1lh krb5kdc[7634](info): set up 2 sockets
Sep 25 22:58:33 dist-test-slave-x1lh krb5kdc[7634](info): commencing operation
krb5kdc: starting...
W20260925 22:58:34.485255 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:58:34 dist-test-slave-x1lh krb5kdc[7634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:58:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:58:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:58:34.641202 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:38881
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:38881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:34.755884 7647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:34.756260 7647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:34.756361 7647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:34.760699 7647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:58:34.760834 7647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:34.760874 7647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:34.760926 7647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:58:34.760963 7647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:58:34.766798 7647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:38881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:38881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:34.768069 7647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:34.769227 7647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:34.778620 7652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:34.778677 7655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:34.778622 7653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:34.778952 7647 server_base.cc:1034] running on GCE node
I20260925 22:58:34.779726 7647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:34.781045 7647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:34.782230 7647 hybrid_clock.cc:648] HybridClock initialized: now 1790377114782193 us; error 55 us; skew 500 ppm
Sep 25 22:58:34 dist-test-slave-x1lh krb5kdc[7634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377114, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:34.791138 7647 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:58:34.792230 7647 webserver.cc:460] Webserver started at http://127.20.114.254:36953/ using document root <none> and password file <none>
I20260925 22:58:34.792953 7647 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:34.793032 7647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:34.793231 7647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:34.795576 7647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "15ae026573564df191737a975f25f3ea"
format_stamp: "Formatted at 2026-09-25 22:58:34 on dist-test-slave-x1lh"
server_key: "7c7302ff45b281e219ea0d6081e319e9"
server_key_iv: "5ccf470c91fd32ce7001c16ccea1d2a9"
server_key_version: "encryptionkey@0"
I20260925 22:58:34.796190 7647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "15ae026573564df191737a975f25f3ea"
format_stamp: "Formatted at 2026-09-25 22:58:34 on dist-test-slave-x1lh"
server_key: "7c7302ff45b281e219ea0d6081e319e9"
server_key_iv: "5ccf470c91fd32ce7001c16ccea1d2a9"
server_key_version: "encryptionkey@0"
I20260925 22:58:34.800717 7647 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:58:34.803820 7662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:34.805564 7647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:34.805699 7647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "15ae026573564df191737a975f25f3ea"
format_stamp: "Formatted at 2026-09-25 22:58:34 on dist-test-slave-x1lh"
server_key: "7c7302ff45b281e219ea0d6081e319e9"
server_key_iv: "5ccf470c91fd32ce7001c16ccea1d2a9"
server_key_version: "encryptionkey@0"
I20260925 22:58:34.805819 7647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:34.826308 7647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:34.830113 7647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:34.830293 7647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:34.838584 7647 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:38881
I20260925 22:58:34.838598 7714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:38881 every 8 connection(s)
I20260925 22:58:34.839721 7647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:58:34.843369 7715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:34.849057 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 7647
I20260925 22:58:34.849218 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:58:34.849525 20939 external_mini_cluster.cc:1428] Setting key 565928d56f98abc833c0274aabc933c3
I20260925 22:58:34.850364 7715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea: Bootstrap starting.
I20260925 22:58:34.853055 7715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:34.853855 7715 log.cc:824] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:34.855942 7715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea: No bootstrap required, opened a new log
Sep 25 22:58:34 dist-test-slave-x1lh krb5kdc[7634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:58:34.859819 7715 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ae026573564df191737a975f25f3ea" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38881 } }
I20260925 22:58:34.860093 7715 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:34.860189 7715 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ae026573564df191737a975f25f3ea, State: Initialized, Role: FOLLOWER
I20260925 22:58:34.860699 7715 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ae026573564df191737a975f25f3ea" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38881 } }
I20260925 22:58:34.860899 7715 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:58:34.861004 7715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:58:34.861129 7715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:34.862118 7715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ae026573564df191737a975f25f3ea" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38881 } }
I20260925 22:58:34.862509 7715 leader_election.cc:302] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15ae026573564df191737a975f25f3ea; no voters:
I20260925 22:58:34.862885 7715 leader_election.cc:288] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:58:34.862964 7720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:34.863339 7720 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea [term 1 LEADER]: Becoming Leader. State: Replica: 15ae026573564df191737a975f25f3ea, State: Running, Role: LEADER
I20260925 22:58:34.864226 7720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ae026573564df191737a975f25f3ea" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38881 } }
I20260925 22:58:34.864523 7715 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:58:34.864809 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:34.851091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52704 (local address 127.20.114.254:38881)
0925 22:58:34.851607 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:34.851624 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:34.851660 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:58:34.852377 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:34.852396 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:34.852747 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:34.853174 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:34.853993 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:34.855064 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:34.856036 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:34.856553 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:34.856752 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:34.859274 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:34.859280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:34.859295 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:34.859326 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:34.861479 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:34.861964 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:34.861970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:34.861976 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:34.862106 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:34.862406 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:34.862409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:34.862411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:34.862967 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:34.863322 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:34.863576 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 22:58:34.863897 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":172,"threads_started":1}
I20260925 22:58:34.865650 7720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15ae026573564df191737a975f25f3ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ae026573564df191737a975f25f3ea" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38881 } } }
I20260925 22:58:34.865785 7720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:34.866003 7720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15ae026573564df191737a975f25f3ea. Latest consensus state: current_term: 1 leader_uuid: "15ae026573564df191737a975f25f3ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ae026573564df191737a975f25f3ea" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38881 } } }
I20260925 22:58:34.866086 7720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:34.866252 7729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:58:34.869263 7729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:58:34.876055 7729 catalog_manager.cc:1347] Generated new cluster ID: fbe70ff7a2bd4f4dac72a4febb8534ef
I20260925 22:58:34.876142 7729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:58:34.908934 7729 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:58:34.909992 7729 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:58:34.923390 7729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea: Generated new TSK 0
I20260925 22:58:34.924373 7729 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.114.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
Entry for principal kudu/127.20.114.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab.
I20260925 22:58:34.962199 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:38881
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:35.080163 7741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:35.080472 7741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:35.080520 7741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:35.085089 7741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:35.085166 7741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:35.085268 7741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:58:35.090968 7741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.193.keytab
--principal=kudu/127.20.114.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:38881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:35.092178 7741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:35.093276 7741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:35.103847 7746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:35.103890 7747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:35.103911 7749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:35.104321 7741 server_base.cc:1034] running on GCE node
I20260925 22:58:35.104768 7741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:35.105543 7741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:35.106730 7741 hybrid_clock.cc:648] HybridClock initialized: now 1790377115106703 us; error 41 us; skew 500 ppm
Sep 25 22:58:35 dist-test-slave-x1lh krb5kdc[7634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377115, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:35.110615 7741 init.cc:377] Logged in from keytab as kudu/127.20.114.193@KRBTEST.COM (short username kudu)
I20260925 22:58:35.112025 7741 webserver.cc:460] Webserver started at http://127.20.114.193:43579/ using document root <none> and password file <none>
I20260925 22:58:35.112692 7741 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:35.112762 7741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:35.113029 7741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:35.115235 7741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "0a44cea3c8df45799f07b07c23e48688"
format_stamp: "Formatted at 2026-09-25 22:58:35 on dist-test-slave-x1lh"
server_key: "ea3cd0850d14b5272f11a3b45a3439e4"
server_key_iv: "85acc1b8306cd0cdcb90a7397ea56dd0"
server_key_version: "encryptionkey@0"
I20260925 22:58:35.115805 7741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "0a44cea3c8df45799f07b07c23e48688"
format_stamp: "Formatted at 2026-09-25 22:58:35 on dist-test-slave-x1lh"
server_key: "ea3cd0850d14b5272f11a3b45a3439e4"
server_key_iv: "85acc1b8306cd0cdcb90a7397ea56dd0"
server_key_version: "encryptionkey@0"
I20260925 22:58:35.120193 7741 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 22:58:35.122987 7756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:35.124385 7741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 22:58:35.124588 7741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "0a44cea3c8df45799f07b07c23e48688"
format_stamp: "Formatted at 2026-09-25 22:58:35 on dist-test-slave-x1lh"
server_key: "ea3cd0850d14b5272f11a3b45a3439e4"
server_key_iv: "85acc1b8306cd0cdcb90a7397ea56dd0"
server_key_version: "encryptionkey@0"
I20260925 22:58:35.124725 7741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:35.150708 7741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:35.154436 7741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:35.154663 7741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:35.155313 7741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:58:35.156210 7741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:58:35.156267 7741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:35.156311 7741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:58:35.156366 7741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:35.166522 7741 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:36429
I20260925 22:58:35.166564 7869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:36429 every 8 connection(s)
I20260925 22:58:35.167523 7741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:58:35.170234 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 7741
I20260925 22:58:35.170396 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:58:35.170701 20939 external_mini_cluster.cc:1428] Setting key c016faaf273e9f0d053b899e701e13ce
Sep 25 22:58:35 dist-test-slave-x1lh krb5kdc[7634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377115, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.193@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.114.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.194@KRBTEST.COM" created.
I20260925 22:58:35.185353 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:35.169675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:48553 (local address 127.20.114.254:38881)
0925 22:58:35.169854 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:35.169859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:35.170793 (+ 934us) server_negotiation.cc:407] Connection header received
0925 22:58:35.172643 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:35.172647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:35.172700 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:35.172800 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:35.174266 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.174854 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.176282 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.176734 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.176898 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:35.180424 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:35.180427 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:35.180429 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:35.180460 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:35.182214 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:35.182976 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:35.182979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:35.182981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:35.183093 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:35.183605 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:35.183608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:35.183609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:35.183856 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:35.183953 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:35.184856 (+ 903us) server_negotiation.cc:299] Negotiation successful
0925 22:58:35.185073 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 22:58:35.186174 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:35.170120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:38881 (local address 127.20.114.193:48553)
0925 22:58:35.170625 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:35.170684 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:35.172388 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:35.173064 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:35.173073 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:35.173454 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:58:35.174049 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:35.174077 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.175012 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.175017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:35.176104 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:35.176115 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.177012 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.177016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:35.177186 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:35.177908 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:58:35.177938 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:58:35.180135 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:58:35.182441 (+ 2306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:58:35.182453 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:58:35.182478 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:58:35.182843 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:58:35.183232 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:58:35.183239 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:58:35.183243 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:58:35.183482 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:58:35.184026 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:58:35.184033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:58:35.184524 (+ 491us) client_negotiation.cc:769] Sending connection context
0925 22:58:35.184757 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 22:58:35.185172 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":149,"threads_started":1}
I20260925 22:58:35.187711 7870 heartbeater.cc:347] Connected to a master server at 127.20.114.254:38881
I20260925 22:58:35.188006 7870 heartbeater.cc:464] Registering TS with master...
I20260925 22:58:35.188573 7870 heartbeater.cc:511] Master 127.20.114.254:38881 requested a full tablet report, sending...
I20260925 22:58:35.190339 7679 ts_manager.cc:194] Registered new tserver with Master: 0a44cea3c8df45799f07b07c23e48688 (127.20.114.193:36429)
I20260925 22:58:35.191542 7679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.193@KRBTEST.COM'} at 127.20.114.193:48553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
Entry for principal kudu/127.20.114.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab.
I20260925 22:58:35.205514 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:38881
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:35.326442 7875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:35.326807 7875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:35.326898 7875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:35.331441 7875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:35.331557 7875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:35.331667 7875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:58:35.337530 7875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.194.keytab
--principal=kudu/127.20.114.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:38881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:35.338757 7875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:35.339936 7875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:35.350620 7881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:35.350610 7883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:35.350610 7880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:35.351047 7875 server_base.cc:1034] running on GCE node
I20260925 22:58:35.351671 7875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:35.352581 7875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:35.353808 7875 hybrid_clock.cc:648] HybridClock initialized: now 1790377115353764 us; error 75 us; skew 500 ppm
Sep 25 22:58:35 dist-test-slave-x1lh krb5kdc[7634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377115, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:35.357388 7875 init.cc:377] Logged in from keytab as kudu/127.20.114.194@KRBTEST.COM (short username kudu)
I20260925 22:58:35.358421 7875 webserver.cc:460] Webserver started at http://127.20.114.194:34209/ using document root <none> and password file <none>
I20260925 22:58:35.359066 7875 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:35.359138 7875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:35.359376 7875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:35.362202 7875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "d93fa21702ce49de9fade695d08a076e"
format_stamp: "Formatted at 2026-09-25 22:58:35 on dist-test-slave-x1lh"
server_key: "3651c77c5031361cfd681f71b74bad91"
server_key_iv: "59ab8c54bd73f1f80961f85a02f9c746"
server_key_version: "encryptionkey@0"
I20260925 22:58:35.363060 7875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "d93fa21702ce49de9fade695d08a076e"
format_stamp: "Formatted at 2026-09-25 22:58:35 on dist-test-slave-x1lh"
server_key: "3651c77c5031361cfd681f71b74bad91"
server_key_iv: "59ab8c54bd73f1f80961f85a02f9c746"
server_key_version: "encryptionkey@0"
I20260925 22:58:35.369005 7875 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 22:58:35.371948 7890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:35.373674 7875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:35.373855 7875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "d93fa21702ce49de9fade695d08a076e"
format_stamp: "Formatted at 2026-09-25 22:58:35 on dist-test-slave-x1lh"
server_key: "3651c77c5031361cfd681f71b74bad91"
server_key_iv: "59ab8c54bd73f1f80961f85a02f9c746"
server_key_version: "encryptionkey@0"
I20260925 22:58:35.374001 7875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:35.392549 7875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:35.396355 7875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:35.396543 7875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:35.397245 7875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:58:35.398123 7875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:58:35.398175 7875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:35.398216 7875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:58:35.398233 7875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:35.408557 7875 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:36571
I20260925 22:58:35.408643 8003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:36571 every 8 connection(s)
I20260925 22:58:35.409634 7875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:58:35.413340 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 7875
I20260925 22:58:35.413481 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:58:35.413805 20939 external_mini_cluster.cc:1428] Setting key 1c7bed567a1b1c36d742355b9d6187bb
Sep 25 22:58:35 dist-test-slave-x1lh krb5kdc[7634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377115, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.194@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.114.195@KRBTEST.COM; defaulting to no policy
I20260925 22:58:35.426352 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:35.412037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:60667 (local address 127.20.114.254:38881)
0925 22:58:35.412189 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:35.412193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:35.413379 (+ 1186us) server_negotiation.cc:407] Connection header received
0925 22:58:35.415022 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:35.415026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:35.415078 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:35.415152 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:35.416462 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.417182 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.418531 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.418967 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.419094 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:35.421903 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:35.421909 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:35.421912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:35.421948 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:35.423853 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:35.424411 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:35.424414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:35.424416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:35.424497 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:35.424985 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:35.424988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:35.424989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:35.425320 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:35.425431 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:35.426059 (+ 628us) server_negotiation.cc:299] Negotiation successful
0925 22:58:35.426179 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.195@KRBTEST.COM" created.
I20260925 22:58:35.427325 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:35.412499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:38881 (local address 127.20.114.194:60667)
0925 22:58:35.413194 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:35.413245 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:35.414787 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:35.415354 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:35.415362 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:35.415769 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:58:35.416296 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:35.416310 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.417338 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.417341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:35.418332 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:35.418394 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.419137 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.419140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:35.419311 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:35.420040 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:58:35.420059 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:58:35.421682 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:58:35.424008 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:58:35.424014 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:58:35.424030 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:58:35.424315 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:58:35.424584 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:58:35.424586 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:58:35.424588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:58:35.424751 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:58:35.425486 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:58:35.425492 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:58:35.425770 (+ 278us) client_negotiation.cc:769] Sending connection context
0925 22:58:35.426002 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 22:58:35.426298 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":148,"threads_started":1}
I20260925 22:58:35.428799 8004 heartbeater.cc:347] Connected to a master server at 127.20.114.254:38881
I20260925 22:58:35.429126 8004 heartbeater.cc:464] Registering TS with master...
I20260925 22:58:35.429867 8004 heartbeater.cc:511] Master 127.20.114.254:38881 requested a full tablet report, sending...
I20260925 22:58:35.431078 7679 ts_manager.cc:194] Registered new tserver with Master: d93fa21702ce49de9fade695d08a076e (127.20.114.194:36571)
I20260925 22:58:35.431697 7679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.194@KRBTEST.COM'} at 127.20.114.194:60667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
Entry for principal kudu/127.20.114.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab.
I20260925 22:58:35.446431 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:38881
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:35.575975 8009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:35.576324 8009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:35.576373 8009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:35.581351 8009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:35.581452 8009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:35.581519 8009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:58:35.587790 8009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.195.keytab
--principal=kudu/127.20.114.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:38881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:35.589121 8009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:35.590299 8009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:35.601248 8017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:35.601263 8014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:35.601267 8009 server_base.cc:1034] running on GCE node
W20260925 22:58:35.601274 8015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:35.602077 8009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:35.602881 8009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:35.604082 8009 hybrid_clock.cc:648] HybridClock initialized: now 1790377115604065 us; error 35 us; skew 500 ppm
Sep 25 22:58:35 dist-test-slave-x1lh krb5kdc[7634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377115, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:35.607774 8009 init.cc:377] Logged in from keytab as kudu/127.20.114.195@KRBTEST.COM (short username kudu)
I20260925 22:58:35.609048 8009 webserver.cc:460] Webserver started at http://127.20.114.195:39881/ using document root <none> and password file <none>
I20260925 22:58:35.609764 8009 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:35.609911 8009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:35.610153 8009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:35.612571 8009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "29019bef199d4fe095ed45650aad50d3"
format_stamp: "Formatted at 2026-09-25 22:58:35 on dist-test-slave-x1lh"
server_key: "0953a5b6771be31c14b3a981d6732a4f"
server_key_iv: "b7f011f0ab84637089ef4e62000699e7"
server_key_version: "encryptionkey@0"
I20260925 22:58:35.613330 8009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "29019bef199d4fe095ed45650aad50d3"
format_stamp: "Formatted at 2026-09-25 22:58:35 on dist-test-slave-x1lh"
server_key: "0953a5b6771be31c14b3a981d6732a4f"
server_key_iv: "b7f011f0ab84637089ef4e62000699e7"
server_key_version: "encryptionkey@0"
I20260925 22:58:35.618168 8009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:58:35.621496 8024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:35.623158 8009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:58:35.623338 8009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "29019bef199d4fe095ed45650aad50d3"
format_stamp: "Formatted at 2026-09-25 22:58:35 on dist-test-slave-x1lh"
server_key: "0953a5b6771be31c14b3a981d6732a4f"
server_key_iv: "b7f011f0ab84637089ef4e62000699e7"
server_key_version: "encryptionkey@0"
I20260925 22:58:35.623514 8009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:35.643239 8009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:35.646710 8009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:35.646888 8009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:35.647455 8009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:58:35.648281 8009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:58:35.648329 8009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:35.648375 8009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:58:35.648391 8009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:35.657510 8009 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:41753
I20260925 22:58:35.657549 8137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:41753 every 8 connection(s)
I20260925 22:58:35.658457 8009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:58:35.664609 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 8009
I20260925 22:58:35.664753 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:58:35.665100 20939 external_mini_cluster.cc:1428] Setting key 23798f9c5d31c9363e9983abfc590065
Sep 25 22:58:35 dist-test-slave-x1lh krb5kdc[7634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377115, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.195@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:58:35.674183 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:35.660486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:43771 (local address 127.20.114.254:38881)
0925 22:58:35.660629 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:35.660632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:35.661978 (+ 1346us) server_negotiation.cc:407] Connection header received
0925 22:58:35.662904 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:35.662908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:35.662965 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:35.663033 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:35.664244 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.665143 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.667003 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.667224 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.667371 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:35.670166 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:35.670170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:35.670172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:35.670202 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:35.671750 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:35.672400 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:35.672403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:35.672405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:35.672497 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:35.672980 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:35.672983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:35.672984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:35.673182 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:35.673272 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:35.673923 (+ 651us) server_negotiation.cc:299] Negotiation successful
0925 22:58:35.674051 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:58:35.675119 8140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:35.660928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:38881 (local address 127.20.114.195:43771)
0925 22:58:35.661815 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:35.661860 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:35.662683 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:35.663204 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:35.663211 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:35.663573 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:58:35.664061 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:35.664075 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.665356 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.665361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:35.666887 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:35.666898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.667421 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.667425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:35.667636 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:35.668227 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:58:35.668249 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:58:35.669980 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:58:35.671909 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:58:35.671915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:58:35.671941 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:58:35.672269 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:58:35.672620 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:58:35.672623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:58:35.672624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:58:35.672804 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:58:35.673324 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:58:35.673332 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:58:35.673622 (+ 290us) client_negotiation.cc:769] Sending connection context
0925 22:58:35.673795 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 22:58:35.674123 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":543,"threads_started":1}
I20260925 22:58:35.676467 8138 heartbeater.cc:347] Connected to a master server at 127.20.114.254:38881
I20260925 22:58:35.676748 8138 heartbeater.cc:464] Registering TS with master...
I20260925 22:58:35.677368 8138 heartbeater.cc:511] Master 127.20.114.254:38881 requested a full tablet report, sending...
I20260925 22:58:35.678701 7674 ts_manager.cc:194] Registered new tserver with Master: 29019bef199d4fe095ed45650aad50d3 (127.20.114.195:41753)
I20260925 22:58:35.679559 7674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.114.195@KRBTEST.COM'} at 127.20.114.195:43771
I20260925 22:58:35.679657 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:58:35.690028 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:35.681919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52708 (local address 127.20.114.254:38881)
0925 22:58:35.682058 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:35.682061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:35.682248 (+ 187us) server_negotiation.cc:407] Connection header received
0925 22:58:35.682361 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:35.682363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:35.682399 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:35.682467 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:35.683229 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.683769 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.684656 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.684999 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.685093 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:35.686577 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:35.686581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:35.686583 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:35.686608 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:35.688116 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:35.688602 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:35.688605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:35.688606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:35.688699 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:35.689124 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:35.689127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:35.689128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:35.689413 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:35.689532 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:35.689767 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 22:58:35.689891 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:58:35.693630 7674 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:58:35.697074 7674 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:58:35.713847 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:35.706598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:36571 (local address 127.0.0.1:55834)
0925 22:58:35.707621 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:35.707679 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:35.707812 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:35.708591 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:35.708594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:35.708606 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:35.708799 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:35.708811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.710402 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.710406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:35.711824 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:35.711833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.713275 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.713282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:35.713399 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:35.713417 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 22:58:35.713537 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 22:58:35.713682 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":82,"spinlock_wait_cycles":15744,"thread_start_us":100,"threads_started":1}
I20260925 22:58:35.713853 8151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:35.706746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55834 (local address 127.20.114.194:36571)
0925 22:58:35.708111 (+ 1365us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:35.708123 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:35.708145 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:58:35.708223 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:35.708229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:35.708410 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:35.708716 (+ 306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:35.709001 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.710239 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.711950 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.712844 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.713040 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:35.713204 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:35.713577 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 22:58:35.713682 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1252,"thread_start_us":127,"threads_started":1}
I20260925 22:58:35.713953 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:35.707489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41362 (local address 127.20.114.193:36429)
0925 22:58:35.708476 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:35.708483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:35.708507 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:58:35.708581 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:35.708586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:35.708783 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:35.709015 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:35.709507 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.710542 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.712181 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.713042 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.713156 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:35.713259 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:35.713655 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 22:58:35.713771 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":846,"thread_start_us":105,"threads_started":1}
I20260925 22:58:35.714277 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:35.706883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:36429 (local address 127.0.0.1:41362)
0925 22:58:35.708043 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:35.708056 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:35.708132 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:35.709196 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:35.709203 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:35.709223 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:35.709360 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:35.709367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.710676 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.710679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:35.712055 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:35.712062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.713169 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.713174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:35.713351 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:35.713403 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 22:58:35.713525 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 22:58:35.714152 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102,"mutex_wait_us":671,"thread_start_us":77,"threads_started":1}
I20260925 22:58:35.715689 8149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:35.706598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:41753 (local address 127.0.0.1:44156)
0925 22:58:35.707666 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:35.707679 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:35.707813 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:35.708513 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:35.708522 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:35.708551 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:35.708771 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:35.708780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.710583 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.710586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:35.712041 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:35.712053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.715300 (+ 3247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.715304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:35.715397 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:35.715422 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:58:35.715484 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 22:58:35.715554 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":120,"spinlock_wait_cycles":6528,"thread_start_us":92,"threads_started":1}
I20260925 22:58:35.715772 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:35.707543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44156 (local address 127.20.114.195:41753)
0925 22:58:35.707942 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:35.707956 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:35.707981 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:58:35.708068 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:35.708074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:35.708323 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:35.708512 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:35.708983 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.710239 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.713013 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.715076 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.715285 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:35.715447 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:35.715527 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 22:58:35.715628 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":130,"threads_started":1}
I20260925 22:58:35.717571 7804 tablet_service.cc:1467] Processing CreateTablet for tablet ded090c9f7e4491a9df686f5a5920a68 (DEFAULT_TABLE table=test-table [id=b25ba8c288bd41a0bc9a5da0c888b514]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:58:35.717588 7938 tablet_service.cc:1467] Processing CreateTablet for tablet ded090c9f7e4491a9df686f5a5920a68 (DEFAULT_TABLE table=test-table [id=b25ba8c288bd41a0bc9a5da0c888b514]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:58:35.718713 7804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ded090c9f7e4491a9df686f5a5920a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:35.718720 7938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ded090c9f7e4491a9df686f5a5920a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:35.718696 8072 tablet_service.cc:1467] Processing CreateTablet for tablet ded090c9f7e4491a9df686f5a5920a68 (DEFAULT_TABLE table=test-table [id=b25ba8c288bd41a0bc9a5da0c888b514]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:58:35.719704 8072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ded090c9f7e4491a9df686f5a5920a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:35.725693 8155 tablet_bootstrap.cc:492] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688: Bootstrap starting.
I20260925 22:58:35.727123 8156 tablet_bootstrap.cc:492] T ded090c9f7e4491a9df686f5a5920a68 P d93fa21702ce49de9fade695d08a076e: Bootstrap starting.
I20260925 22:58:35.727460 8157 tablet_bootstrap.cc:492] T ded090c9f7e4491a9df686f5a5920a68 P 29019bef199d4fe095ed45650aad50d3: Bootstrap starting.
I20260925 22:58:35.728853 8155 tablet_bootstrap.cc:654] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:35.729602 8157 tablet_bootstrap.cc:654] T ded090c9f7e4491a9df686f5a5920a68 P 29019bef199d4fe095ed45650aad50d3: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:35.729602 8156 tablet_bootstrap.cc:654] T ded090c9f7e4491a9df686f5a5920a68 P d93fa21702ce49de9fade695d08a076e: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:35.730623 8155 log.cc:824] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:35.731002 8156 log.cc:824] T ded090c9f7e4491a9df686f5a5920a68 P d93fa21702ce49de9fade695d08a076e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:35.731002 8157 log.cc:824] T ded090c9f7e4491a9df686f5a5920a68 P 29019bef199d4fe095ed45650aad50d3: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:35.733034 8155 tablet_bootstrap.cc:492] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688: No bootstrap required, opened a new log
I20260925 22:58:35.733186 8156 tablet_bootstrap.cc:492] T ded090c9f7e4491a9df686f5a5920a68 P d93fa21702ce49de9fade695d08a076e: No bootstrap required, opened a new log
I20260925 22:58:35.733270 8155 ts_tablet_manager.cc:1397] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 22:58:35.733424 8156 ts_tablet_manager.cc:1397] T ded090c9f7e4491a9df686f5a5920a68 P d93fa21702ce49de9fade695d08a076e: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:58:35.734315 8157 tablet_bootstrap.cc:492] T ded090c9f7e4491a9df686f5a5920a68 P 29019bef199d4fe095ed45650aad50d3: No bootstrap required, opened a new log
I20260925 22:58:35.734520 8157 ts_tablet_manager.cc:1397] T ded090c9f7e4491a9df686f5a5920a68 P 29019bef199d4fe095ed45650aad50d3: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 22:58:35.736794 8156 raft_consensus.cc:348] T ded090c9f7e4491a9df686f5a5920a68 P d93fa21702ce49de9fade695d08a076e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93fa21702ce49de9fade695d08a076e" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 36571 } } peers { permanent_uuid: "29019bef199d4fe095ed45650aad50d3" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 41753 } } peers { permanent_uuid: "0a44cea3c8df45799f07b07c23e48688" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36429 } }
I20260925 22:58:35.737161 8156 raft_consensus.cc:374] T ded090c9f7e4491a9df686f5a5920a68 P d93fa21702ce49de9fade695d08a076e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:35.737234 8156 raft_consensus.cc:729] T ded090c9f7e4491a9df686f5a5920a68 P d93fa21702ce49de9fade695d08a076e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d93fa21702ce49de9fade695d08a076e, State: Initialized, Role: FOLLOWER
I20260925 22:58:35.737754 8156 consensus_queue.cc:260] T ded090c9f7e4491a9df686f5a5920a68 P d93fa21702ce49de9fade695d08a076e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93fa21702ce49de9fade695d08a076e" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 36571 } } peers { permanent_uuid: "29019bef199d4fe095ed45650aad50d3" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 41753 } } peers { permanent_uuid: "0a44cea3c8df45799f07b07c23e48688" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36429 } }
I20260925 22:58:35.738029 8155 raft_consensus.cc:348] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93fa21702ce49de9fade695d08a076e" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 36571 } } peers { permanent_uuid: "29019bef199d4fe095ed45650aad50d3" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 41753 } } peers { permanent_uuid: "0a44cea3c8df45799f07b07c23e48688" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36429 } }
I20260925 22:58:35.738332 8155 raft_consensus.cc:374] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:35.738404 8155 raft_consensus.cc:729] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a44cea3c8df45799f07b07c23e48688, State: Initialized, Role: FOLLOWER
I20260925 22:58:35.738641 8004 heartbeater.cc:503] Master 127.20.114.254:38881 was elected leader, sending a full tablet report...
I20260925 22:58:35.738566 8157 raft_consensus.cc:348] T ded090c9f7e4491a9df686f5a5920a68 P 29019bef199d4fe095ed45650aad50d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93fa21702ce49de9fade695d08a076e" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 36571 } } peers { permanent_uuid: "29019bef199d4fe095ed45650aad50d3" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 41753 } } peers { permanent_uuid: "0a44cea3c8df45799f07b07c23e48688" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36429 } }
I20260925 22:58:35.738801 8157 raft_consensus.cc:374] T ded090c9f7e4491a9df686f5a5920a68 P 29019bef199d4fe095ed45650aad50d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:35.738862 8157 raft_consensus.cc:729] T ded090c9f7e4491a9df686f5a5920a68 P 29019bef199d4fe095ed45650aad50d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29019bef199d4fe095ed45650aad50d3, State: Initialized, Role: FOLLOWER
I20260925 22:58:35.738925 8156 ts_tablet_manager.cc:1428] T ded090c9f7e4491a9df686f5a5920a68 P d93fa21702ce49de9fade695d08a076e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:58:35.738893 8155 consensus_queue.cc:260] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93fa21702ce49de9fade695d08a076e" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 36571 } } peers { permanent_uuid: "29019bef199d4fe095ed45650aad50d3" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 41753 } } peers { permanent_uuid: "0a44cea3c8df45799f07b07c23e48688" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36429 } }
I20260925 22:58:35.739306 8157 consensus_queue.cc:260] T ded090c9f7e4491a9df686f5a5920a68 P 29019bef199d4fe095ed45650aad50d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93fa21702ce49de9fade695d08a076e" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 36571 } } peers { permanent_uuid: "29019bef199d4fe095ed45650aad50d3" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 41753 } } peers { permanent_uuid: "0a44cea3c8df45799f07b07c23e48688" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36429 } }
I20260925 22:58:35.739595 7870 heartbeater.cc:503] Master 127.20.114.254:38881 was elected leader, sending a full tablet report...
I20260925 22:58:35.739907 8155 ts_tablet_manager.cc:1428] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:58:35.740958 8138 heartbeater.cc:503] Master 127.20.114.254:38881 was elected leader, sending a full tablet report...
I20260925 22:58:35.741322 8157 ts_tablet_manager.cc:1428] T ded090c9f7e4491a9df686f5a5920a68 P 29019bef199d4fe095ed45650aad50d3: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 22:58:35.907543 8162 raft_consensus.cc:482] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:58:35.907835 8162 raft_consensus.cc:504] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93fa21702ce49de9fade695d08a076e" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 36571 } } peers { permanent_uuid: "29019bef199d4fe095ed45650aad50d3" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 41753 } } peers { permanent_uuid: "0a44cea3c8df45799f07b07c23e48688" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36429 } }
I20260925 22:58:35.909085 8162 leader_election.cc:288] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d93fa21702ce49de9fade695d08a076e (127.20.114.194:36571), 29019bef199d4fe095ed45650aad50d3 (127.20.114.195:41753)
W20260925 22:58:35.909627 8139 tablet.cc:2369] T ded090c9f7e4491a9df686f5a5920a68 P 29019bef199d4fe095ed45650aad50d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:58:35.911013 8005 tablet.cc:2369] T ded090c9f7e4491a9df686f5a5920a68 P d93fa21702ce49de9fade695d08a076e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:58:35.915196 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:35.909657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:32997 (local address 127.20.114.195:41753)
0925 22:58:35.909807 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:35.909812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:35.910305 (+ 493us) server_negotiation.cc:407] Connection header received
0925 22:58:35.910531 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:35.910536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:35.910588 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:35.910685 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:35.910974 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.911681 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.913678 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.914414 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.914566 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:35.914627 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:35.914962 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 22:58:35.915047 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 22:58:35.915196 8151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:35.909678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:47377 (local address 127.20.114.194:36571)
0925 22:58:35.909829 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:35.909833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:35.910137 (+ 304us) server_negotiation.cc:407] Connection header received
0925 22:58:35.910367 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:35.910371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:35.910428 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:35.910521 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:35.910827 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.911647 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.913678 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.914414 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.914566 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:35.914627 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:35.914962 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 22:58:35.915047 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 22:58:35.915258 8164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:35.909472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:36571 (local address 127.20.114.193:47377)
0925 22:58:35.910034 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:35.910052 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:35.910231 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:35.910535 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:35.910538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:35.910555 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:35.910699 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:35.910705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.911868 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.911870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:35.913499 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:35.913513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.914571 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.914575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:35.914827 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:35.914844 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 22:58:35.914910 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 22:58:35.915033 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":250,"thread_start_us":132,"threads_started":1}
I20260925 22:58:35.915258 8165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:35.909472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:41753 (local address 127.20.114.193:32997)
0925 22:58:35.910198 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:35.910213 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:35.910412 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:35.910697 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:35.910700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:35.910718 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:35.910837 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:35.910847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.911813 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.911817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:35.913499 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:35.913513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.914571 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.914575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:35.914770 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:35.914827 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 22:58:35.914910 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 22:58:35.915033 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":612,"thread_start_us":142,"threads_started":1}
I20260925 22:58:35.916160 7958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded090c9f7e4491a9df686f5a5920a68" candidate_uuid: "0a44cea3c8df45799f07b07c23e48688" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93fa21702ce49de9fade695d08a076e" is_pre_election: true
I20260925 22:58:35.916431 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded090c9f7e4491a9df686f5a5920a68" candidate_uuid: "0a44cea3c8df45799f07b07c23e48688" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29019bef199d4fe095ed45650aad50d3" is_pre_election: true
I20260925 22:58:35.916544 7958 raft_consensus.cc:2454] T ded090c9f7e4491a9df686f5a5920a68 P d93fa21702ce49de9fade695d08a076e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a44cea3c8df45799f07b07c23e48688 in term 0.
I20260925 22:58:35.916671 8092 raft_consensus.cc:2454] T ded090c9f7e4491a9df686f5a5920a68 P 29019bef199d4fe095ed45650aad50d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a44cea3c8df45799f07b07c23e48688 in term 0.
I20260925 22:58:35.917225 7760 leader_election.cc:302] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0a44cea3c8df45799f07b07c23e48688, 29019bef199d4fe095ed45650aad50d3, d93fa21702ce49de9fade695d08a076e; no voters:
I20260925 22:58:35.917567 8162 raft_consensus.cc:2781] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:58:35.917660 8162 raft_consensus.cc:482] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:58:35.917706 8162 raft_consensus.cc:3037] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688 [term 0 FOLLOWER]: Advancing to term 1
W20260925 22:58:35.919000 7871 tablet.cc:2369] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:58:35.918951 8162 raft_consensus.cc:504] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93fa21702ce49de9fade695d08a076e" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 36571 } } peers { permanent_uuid: "29019bef199d4fe095ed45650aad50d3" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 41753 } } peers { permanent_uuid: "0a44cea3c8df45799f07b07c23e48688" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36429 } }
I20260925 22:58:35.919370 8162 leader_election.cc:288] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688 [CANDIDATE]: Term 1 election: Requested vote from peers d93fa21702ce49de9fade695d08a076e (127.20.114.194:36571), 29019bef199d4fe095ed45650aad50d3 (127.20.114.195:41753)
I20260925 22:58:35.919801 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded090c9f7e4491a9df686f5a5920a68" candidate_uuid: "0a44cea3c8df45799f07b07c23e48688" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29019bef199d4fe095ed45650aad50d3"
I20260925 22:58:35.919956 8092 raft_consensus.cc:3037] T ded090c9f7e4491a9df686f5a5920a68 P 29019bef199d4fe095ed45650aad50d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:35.920001 7958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded090c9f7e4491a9df686f5a5920a68" candidate_uuid: "0a44cea3c8df45799f07b07c23e48688" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93fa21702ce49de9fade695d08a076e"
I20260925 22:58:35.920140 7958 raft_consensus.cc:3037] T ded090c9f7e4491a9df686f5a5920a68 P d93fa21702ce49de9fade695d08a076e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:35.921469 7958 raft_consensus.cc:2454] T ded090c9f7e4491a9df686f5a5920a68 P d93fa21702ce49de9fade695d08a076e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a44cea3c8df45799f07b07c23e48688 in term 1.
I20260925 22:58:35.921469 8092 raft_consensus.cc:2454] T ded090c9f7e4491a9df686f5a5920a68 P 29019bef199d4fe095ed45650aad50d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a44cea3c8df45799f07b07c23e48688 in term 1.
I20260925 22:58:35.921892 7760 leader_election.cc:302] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a44cea3c8df45799f07b07c23e48688, 29019bef199d4fe095ed45650aad50d3; no voters:
I20260925 22:58:35.922137 8162 raft_consensus.cc:2781] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:35.922422 8162 raft_consensus.cc:686] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688 [term 1 LEADER]: Becoming Leader. State: Replica: 0a44cea3c8df45799f07b07c23e48688, State: Running, Role: LEADER
I20260925 22:58:35.922776 8162 consensus_queue.cc:237] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93fa21702ce49de9fade695d08a076e" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 36571 } } peers { permanent_uuid: "29019bef199d4fe095ed45650aad50d3" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 41753 } } peers { permanent_uuid: "0a44cea3c8df45799f07b07c23e48688" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36429 } }
I20260925 22:58:35.926219 7674 catalog_manager.cc:5495] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a44cea3c8df45799f07b07c23e48688 (127.20.114.193). New cstate: current_term: 1 leader_uuid: "0a44cea3c8df45799f07b07c23e48688" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93fa21702ce49de9fade695d08a076e" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29019bef199d4fe095ed45650aad50d3" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 41753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a44cea3c8df45799f07b07c23e48688" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 36429 } health_report { overall_health: HEALTHY } } }
I20260925 22:58:35.953724 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:35.949593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41366 (local address 127.20.114.193:36429)
0925 22:58:35.949758 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:35.949761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:35.949777 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:58:35.949874 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:35.949877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:35.949945 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:35.950060 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:58:35.950431 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.950984 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.951978 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:35.952210 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:35.952306 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:35.952695 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:58:35.953205 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:58:35.953305 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:35.953460 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 22:58:35.953524 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:58:35.960456 7958 raft_consensus.cc:1261] T ded090c9f7e4491a9df686f5a5920a68 P d93fa21702ce49de9fade695d08a076e [term 1 FOLLOWER]: Refusing update from remote peer 0a44cea3c8df45799f07b07c23e48688: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:58:35.960456 8092 raft_consensus.cc:1261] T ded090c9f7e4491a9df686f5a5920a68 P 29019bef199d4fe095ed45650aad50d3 [term 1 FOLLOWER]: Refusing update from remote peer 0a44cea3c8df45799f07b07c23e48688: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:58:35.961380 8162 consensus_queue.cc:1035] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d93fa21702ce49de9fade695d08a076e" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:58:35.961655 8166 consensus_queue.cc:1035] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29019bef199d4fe095ed45650aad50d3" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 41753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:58:35.970239 8169 mvcc.cc:205] Tried to move back new op lower bound from 7333384666965430272 to 7333384666822582272. Current Snapshot: MvccSnapshot[applied={T|T < 7333384666965430272}]
I20260925 22:58:35.972302 8171 mvcc.cc:205] Tried to move back new op lower bound from 7333384666965430272 to 7333384666822582272. Current Snapshot: MvccSnapshot[applied={T|T < 7333384666965430272}]
I20260925 22:58:35.980152 7674 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:58:35.980382 7674 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:58:35.983534 7674 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15ae026573564df191737a975f25f3ea: Sending DeleteTablet for 3 replicas of tablet ded090c9f7e4491a9df686f5a5920a68
I20260925 22:58:35.984534 8072 tablet_service.cc:1514] Processing DeleteTablet for tablet ded090c9f7e4491a9df686f5a5920a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:58:35 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:44156
I20260925 22:58:35.984834 7804 tablet_service.cc:1514] Processing DeleteTablet for tablet ded090c9f7e4491a9df686f5a5920a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:58:35 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:41362
I20260925 22:58:35.985221 7938 tablet_service.cc:1514] Processing DeleteTablet for tablet ded090c9f7e4491a9df686f5a5920a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:58:35 UTC) from {username='kudu', principal='kudu/127.20.114.254@KRBTEST.COM'} at 127.0.0.1:55834
I20260925 22:58:35.985702 8180 tablet_replica.cc:330] T ded090c9f7e4491a9df686f5a5920a68 P 0a44cea3c8df45799f07b07c23e48688: stopping tablet replica
I20260925 22:58:35.985746 8179 tablet_replica.cc:330] T ded090c9f7e4491a9df686f5a5920a68 P 29019bef199d4fe095ed45650aad50d3: stopping tablet replica
I20260925 22:58:35.985806 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 7741
I20260925 22:58:35.987382 8181 tablet_replica.cc:330] T ded090c9f7e4491a9df686f5a5920a68 P d93fa21702ce49de9fade695d08a076e: stopping tablet replica
I20260925 22:58:35.988181 8181 raft_consensus.cc:2229] T ded090c9f7e4491a9df686f5a5920a68 P d93fa21702ce49de9fade695d08a076e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:58:35.988572 8179 raft_consensus.cc:2229] T ded090c9f7e4491a9df686f5a5920a68 P 29019bef199d4fe095ed45650aad50d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:58:35.988708 8181 raft_consensus.cc:2258] T ded090c9f7e4491a9df686f5a5920a68 P d93fa21702ce49de9fade695d08a076e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:58:35.989019 8179 raft_consensus.cc:2258] T ded090c9f7e4491a9df686f5a5920a68 P 29019bef199d4fe095ed45650aad50d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:58:35.990806 8181 ts_tablet_manager.cc:1905] T ded090c9f7e4491a9df686f5a5920a68 P d93fa21702ce49de9fade695d08a076e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:58:35.990806 8179 ts_tablet_manager.cc:1905] T ded090c9f7e4491a9df686f5a5920a68 P 29019bef199d4fe095ed45650aad50d3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 22:58:35.992902 7664 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.114.193:36429 (error 108)
I20260925 22:58:35.994091 8149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:35.993722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:36429 (local address 127.0.0.1:41370)
0925 22:58:35.993895 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:35.993979 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.193:36429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 22:58:35.994076 8181 ts_tablet_manager.cc:1918] T ded090c9f7e4491a9df686f5a5920a68 P d93fa21702ce49de9fade695d08a076e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:58:35.994076 8179 ts_tablet_manager.cc:1918] T ded090c9f7e4491a9df686f5a5920a68 P 29019bef199d4fe095ed45650aad50d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:58:35.994189 8181 log.cc:1196] T ded090c9f7e4491a9df686f5a5920a68 P d93fa21702ce49de9fade695d08a076e: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/ded090c9f7e4491a9df686f5a5920a68
I20260925 22:58:35.994189 8179 log.cc:1196] T ded090c9f7e4491a9df686f5a5920a68 P 29019bef199d4fe095ed45650aad50d3: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/ded090c9f7e4491a9df686f5a5920a68
I20260925 22:58:35.994225 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 7875
I20260925 22:58:35.994551 8181 ts_tablet_manager.cc:1939] T ded090c9f7e4491a9df686f5a5920a68 P d93fa21702ce49de9fade695d08a076e: Deleting consensus metadata
I20260925 22:58:35.994551 8179 ts_tablet_manager.cc:1939] T ded090c9f7e4491a9df686f5a5920a68 P 29019bef199d4fe095ed45650aad50d3: Deleting consensus metadata
I20260925 22:58:35.995909 7666 catalog_manager.cc:4841] TS 29019bef199d4fe095ed45650aad50d3 (127.20.114.195:41753): tablet ded090c9f7e4491a9df686f5a5920a68 (table test-table [id=b25ba8c288bd41a0bc9a5da0c888b514]) successfully deleted
W20260925 22:58:35.997138 7664 catalog_manager.cc:4584] TS 0a44cea3c8df45799f07b07c23e48688 (127.20.114.193:36429): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ded090c9f7e4491a9df686f5a5920a68: Network error: Client connection negotiation failed: client connection to 127.20.114.193:36429: connect: Connection refused (error 111)
I20260925 22:58:36.001513 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 8009
I20260925 22:58:36.002743 8149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:36.001836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:36571 (local address 127.0.0.1:55838)
0925 22:58:36.002540 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:36.002621 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.194:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583}
I20260925 22:58:36.007718 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 7647
2026-09-25T22:58:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2574 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T22:58:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:58:36Z Disabled control of system clock
I20260925 22:58:36.039702 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:39613
--webserver_interface=127.20.114.254
--webserver_port=0
--builtin_ntp_servers=127.20.114.212:35081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:39613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:58:36.159122 8186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:36.159435 8186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:36.159472 8186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:36.163661 8186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:58:36.163748 8186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:36.163764 8186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:36.163782 8186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:58:36.163797 8186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:58:36.169129 8186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:39613
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:39613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:36.170351 8186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:36.171840 8186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:36.181598 8194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:36.181599 8191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:36.181620 8192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:36.182025 8186 server_base.cc:1034] running on GCE node
I20260925 22:58:36.182742 8186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:36.184026 8186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:36.185351 8186 hybrid_clock.cc:648] HybridClock initialized: now 1790377116185318 us; error 40 us; skew 500 ppm
I20260925 22:58:36.187554 8186 webserver.cc:460] Webserver started at http://127.20.114.254:37653/ using document root <none> and password file <none>
I20260925 22:58:36.188165 8186 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:36.188234 8186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:36.188417 8186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:36.190927 8186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "9b657b96f7464cf6b23a3bc6b179a976"
format_stamp: "Formatted at 2026-09-25 22:58:36 on dist-test-slave-x1lh"
server_key: "8a87f6c02cccebbc2f7e79a21c3402e1"
server_key_iv: "06bc2bea6e287578ff3c8bb98512c425"
server_key_version: "encryptionkey@0"
I20260925 22:58:36.191533 8186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "9b657b96f7464cf6b23a3bc6b179a976"
format_stamp: "Formatted at 2026-09-25 22:58:36 on dist-test-slave-x1lh"
server_key: "8a87f6c02cccebbc2f7e79a21c3402e1"
server_key_iv: "06bc2bea6e287578ff3c8bb98512c425"
server_key_version: "encryptionkey@0"
I20260925 22:58:36.195828 8186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:58:36.198931 8200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:36.200505 8186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:58:36.200764 8186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "9b657b96f7464cf6b23a3bc6b179a976"
format_stamp: "Formatted at 2026-09-25 22:58:36 on dist-test-slave-x1lh"
server_key: "8a87f6c02cccebbc2f7e79a21c3402e1"
server_key_iv: "06bc2bea6e287578ff3c8bb98512c425"
server_key_version: "encryptionkey@0"
I20260925 22:58:36.200996 8186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:36.215478 8186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:36.216250 8186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:36.216490 8186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:36.223733 8186 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:39613
I20260925 22:58:36.223783 8252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:39613 every 8 connection(s)
I20260925 22:58:36.224835 8186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:58:36.226699 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 8186
I20260925 22:58:36.226845 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:58:36.227187 20939 external_mini_cluster.cc:1428] Setting key a0addcea06e6c19605545388361e28cb
I20260925 22:58:36.228730 8253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:36.235323 8253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976: Bootstrap starting.
I20260925 22:58:36.236927 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:36.228733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52050 (local address 127.20.114.254:39613)
0925 22:58:36.229527 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:36.229545 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:36.229581 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:58:36.230241 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:36.230257 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:36.230665 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:36.231013 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:36.231454 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.232455 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.233472 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.233977 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.234158 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:36.235066 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:36.235071 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:36.235086 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:36.235546 (+ 460us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:36.235606 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:36.235627 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:36.235733 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:36.235854 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 22:58:36.236132 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":164,"threads_started":1}
I20260925 22:58:36.237902 8253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:36.238824 8253 log.cc:824] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:36.241078 8253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976: No bootstrap required, opened a new log
I20260925 22:58:36.245203 8253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b657b96f7464cf6b23a3bc6b179a976" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 39613 } }
I20260925 22:58:36.245554 8253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:36.245623 8253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b657b96f7464cf6b23a3bc6b179a976, State: Initialized, Role: FOLLOWER
I20260925 22:58:36.246160 8253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b657b96f7464cf6b23a3bc6b179a976" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 39613 } }
I20260925 22:58:36.246302 8253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:58:36.246387 8253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:58:36.246492 8253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:36.247510 8253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b657b96f7464cf6b23a3bc6b179a976" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 39613 } }
I20260925 22:58:36.247905 8253 leader_election.cc:302] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b657b96f7464cf6b23a3bc6b179a976; no voters:
I20260925 22:58:36.248255 8253 leader_election.cc:288] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:58:36.248502 8258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:36.248821 8258 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976 [term 1 LEADER]: Becoming Leader. State: Replica: 9b657b96f7464cf6b23a3bc6b179a976, State: Running, Role: LEADER
I20260925 22:58:36.249181 8258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b657b96f7464cf6b23a3bc6b179a976" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 39613 } }
I20260925 22:58:36.249681 8253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:58:36.251235 8259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b657b96f7464cf6b23a3bc6b179a976" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b657b96f7464cf6b23a3bc6b179a976" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 39613 } } }
I20260925 22:58:36.251389 8259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:36.251909 8260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b657b96f7464cf6b23a3bc6b179a976. Latest consensus state: current_term: 1 leader_uuid: "9b657b96f7464cf6b23a3bc6b179a976" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b657b96f7464cf6b23a3bc6b179a976" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 39613 } } }
I20260925 22:58:36.252406 8260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976 [sys.catalog]: This master's current role is: LEADER
W20260925 22:58:36.254738 8273 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:58:36.254834 8273 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:58:36.254926 8270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:58:36.255694 8270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:58:36.261775 8270 catalog_manager.cc:1347] Generated new cluster ID: bde7430ca1ca4a9a9684349f02824bdf
I20260925 22:58:36.261894 8270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:58:36.273299 8270 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:58:36.274420 8270 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:58:36.282280 8270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976: Generated new TSK 0
I20260925 22:58:36.283244 8270 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:58:36.286913 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.193:0
--local_ip_for_outbound_sockets=127.20.114.193
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:39613
--builtin_ntp_servers=127.20.114.212:35081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:58:36.432641 8277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:36.433003 8277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:36.433048 8277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:36.437465 8277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:36.437536 8277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:36.437597 8277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.193
I20260925 22:58:36.442687 8277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.114.193
--webserver_port=0
--tserver_master_addrs=127.20.114.254:39613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.193
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:36.443948 8277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:36.445562 8277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:36.457194 8285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:36.457564 8277 server_base.cc:1034] running on GCE node
W20260925 22:58:36.457401 8283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:36.457427 8282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:36.458413 8277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:36.459170 8277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:36.460388 8277 hybrid_clock.cc:648] HybridClock initialized: now 1790377116460356 us; error 48 us; skew 500 ppm
I20260925 22:58:36.462651 8277 webserver.cc:460] Webserver started at http://127.20.114.193:43199/ using document root <none> and password file <none>
I20260925 22:58:36.463322 8277 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:36.463420 8277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:36.463665 8277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:36.465991 8277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-0/data/instance:
uuid: "2b5d54dcb75c46aaa07077afb4a6c120"
format_stamp: "Formatted at 2026-09-25 22:58:36 on dist-test-slave-x1lh"
server_key: "49d5f921e3d675d95cb4941df842d44b"
server_key_iv: "77ddaa24e3a2ef23f1f97f6c68237b79"
server_key_version: "encryptionkey@0"
I20260925 22:58:36.466615 8277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance:
uuid: "2b5d54dcb75c46aaa07077afb4a6c120"
format_stamp: "Formatted at 2026-09-25 22:58:36 on dist-test-slave-x1lh"
server_key: "49d5f921e3d675d95cb4941df842d44b"
server_key_iv: "77ddaa24e3a2ef23f1f97f6c68237b79"
server_key_version: "encryptionkey@0"
I20260925 22:58:36.470847 8277 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:58:36.473681 8291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:36.475296 8277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:36.475471 8277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-0/wal
uuid: "2b5d54dcb75c46aaa07077afb4a6c120"
format_stamp: "Formatted at 2026-09-25 22:58:36 on dist-test-slave-x1lh"
server_key: "49d5f921e3d675d95cb4941df842d44b"
server_key_iv: "77ddaa24e3a2ef23f1f97f6c68237b79"
server_key_version: "encryptionkey@0"
I20260925 22:58:36.475616 8277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:36.491377 8277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:36.492295 8277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:36.492552 8277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:36.493382 8277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:58:36.494496 8277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:58:36.494577 8277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:36.494658 8277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:58:36.494717 8277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:36.505223 8277 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.193:39423
I20260925 22:58:36.505316 8404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.193:39423 every 8 connection(s)
I20260925 22:58:36.506434 8277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-0/data/info.pb
I20260925 22:58:36.515784 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 8277
I20260925 22:58:36.516001 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-0/wal/instance
I20260925 22:58:36.516309 20939 external_mini_cluster.cc:1428] Setting key 63ffd30bc9fc5ff3769ebe37d268fe61
I20260925 22:58:36.516855 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:36.508601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:43509 (local address 127.20.114.254:39613)
0925 22:58:36.508751 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:36.508754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:36.510062 (+ 1308us) server_negotiation.cc:407] Connection header received
0925 22:58:36.511611 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:36.511615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:36.511689 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:36.511800 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:36.512777 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.513392 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.514596 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.514838 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.514951 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:36.516279 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:36.516283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:36.516286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:36.516309 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:36.516331 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:36.516340 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:36.516417 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:36.516600 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 22:58:36.516697 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:58:36.517901 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:36.509073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:39613 (local address 127.20.114.193:43509)
0925 22:58:36.509876 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:36.509923 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:36.511353 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:36.511997 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:36.512005 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:36.512096 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:58:36.512611 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:36.512624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.513535 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.513538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:36.514462 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:36.514471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.515016 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.515021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:36.515191 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:36.515912 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:58:36.515913 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:58:36.515958 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:58:36.516076 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:58:36.516431 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:58:36.516443 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:58:36.516449 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:58:36.516599 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 22:58:36.516936 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":161,"threads_started":1}
I20260925 22:58:36.518668 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.194:0
--local_ip_for_outbound_sockets=127.20.114.194
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:39613
--builtin_ntp_servers=127.20.114.212:35081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:58:36.519409 8405 heartbeater.cc:347] Connected to a master server at 127.20.114.254:39613
I20260925 22:58:36.519709 8405 heartbeater.cc:464] Registering TS with master...
I20260925 22:58:36.520541 8405 heartbeater.cc:511] Master 127.20.114.254:39613 requested a full tablet report, sending...
I20260925 22:58:36.522588 8217 ts_manager.cc:194] Registered new tserver with Master: 2b5d54dcb75c46aaa07077afb4a6c120 (127.20.114.193:39423)
I20260925 22:58:36.524430 8217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.114.193:43509
W20260925 22:58:36.644652 8408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:36.645010 8408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:36.645054 8408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:36.649398 8408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:36.649477 8408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:36.649539 8408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.194
I20260925 22:58:36.654970 8408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.114.194
--webserver_port=0
--tserver_master_addrs=127.20.114.254:39613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.194
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:36.656165 8408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:36.657706 8408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:36.668164 8413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:36.668304 8416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:36.668219 8408 server_base.cc:1034] running on GCE node
W20260925 22:58:36.668226 8414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:36.668814 8408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:36.669618 8408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:36.670840 8408 hybrid_clock.cc:648] HybridClock initialized: now 1790377116670837 us; error 60 us; skew 500 ppm
I20260925 22:58:36.673056 8408 webserver.cc:460] Webserver started at http://127.20.114.194:34715/ using document root <none> and password file <none>
I20260925 22:58:36.673677 8408 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:36.673744 8408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:36.673918 8408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:36.676118 8408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-1/data/instance:
uuid: "58f8176383e44d48b77acf61bde3f5cd"
format_stamp: "Formatted at 2026-09-25 22:58:36 on dist-test-slave-x1lh"
server_key: "1f023d1aa931528f5afc604532a5f47a"
server_key_iv: "985cff3dc9f6b20bdfba21ba8c19d1c2"
server_key_version: "encryptionkey@0"
I20260925 22:58:36.676751 8408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance:
uuid: "58f8176383e44d48b77acf61bde3f5cd"
format_stamp: "Formatted at 2026-09-25 22:58:36 on dist-test-slave-x1lh"
server_key: "1f023d1aa931528f5afc604532a5f47a"
server_key_iv: "985cff3dc9f6b20bdfba21ba8c19d1c2"
server_key_version: "encryptionkey@0"
I20260925 22:58:36.681021 8408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 22:58:36.684010 8422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:36.685572 8408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:36.685740 8408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-1/wal
uuid: "58f8176383e44d48b77acf61bde3f5cd"
format_stamp: "Formatted at 2026-09-25 22:58:36 on dist-test-slave-x1lh"
server_key: "1f023d1aa931528f5afc604532a5f47a"
server_key_iv: "985cff3dc9f6b20bdfba21ba8c19d1c2"
server_key_version: "encryptionkey@0"
I20260925 22:58:36.685899 8408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:36.701596 8408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:36.702378 8408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:36.702600 8408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:36.703260 8408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:58:36.704345 8408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:58:36.704419 8408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:36.704494 8408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:58:36.704543 8408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:36.714648 8408 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.194:42261
I20260925 22:58:36.714671 8535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.194:42261 every 8 connection(s)
I20260925 22:58:36.715767 8408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-1/data/info.pb
I20260925 22:58:36.717329 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 8408
I20260925 22:58:36.717474 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-1/wal/instance
I20260925 22:58:36.717759 20939 external_mini_cluster.cc:1428] Setting key 35281730831b78a570d64a6f188fde50
I20260925 22:58:36.720094 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.114.195:0
--local_ip_for_outbound_sockets=127.20.114.195
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:39613
--builtin_ntp_servers=127.20.114.212:35081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:58:36.728089 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:36.718050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.194:40745 (local address 127.20.114.254:39613)
0925 22:58:36.718200 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:36.718204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:36.719656 (+ 1452us) server_negotiation.cc:407] Connection header received
0925 22:58:36.721866 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:36.721873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:36.721950 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:36.722068 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:36.723259 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.724161 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.725636 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.726078 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.726219 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:36.727426 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:36.727432 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:36.727438 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:36.727459 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:36.727490 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:36.727502 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:36.727595 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:36.727767 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 22:58:36.727859 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:58:36.729249 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:36.718565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:39613 (local address 127.20.114.194:40745)
0925 22:58:36.719467 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:36.719528 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:36.721532 (+ 2004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:36.722307 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:36.722315 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:36.722421 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:58:36.723079 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:36.723095 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.724368 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.724375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:36.725496 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:36.725508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.726261 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.726263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:36.726420 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:36.727103 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:58:36.727105 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:58:36.727153 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:58:36.727230 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:58:36.727615 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:58:36.727622 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:58:36.727627 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:58:36.727731 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 22:58:36.728034 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":168,"threads_started":1}
I20260925 22:58:36.730813 8536 heartbeater.cc:347] Connected to a master server at 127.20.114.254:39613
I20260925 22:58:36.731135 8536 heartbeater.cc:464] Registering TS with master...
I20260925 22:58:36.731927 8536 heartbeater.cc:511] Master 127.20.114.254:39613 requested a full tablet report, sending...
I20260925 22:58:36.733258 8217 ts_manager.cc:194] Registered new tserver with Master: 58f8176383e44d48b77acf61bde3f5cd (127.20.114.194:42261)
I20260925 22:58:36.733904 8217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.114.194:40745
W20260925 22:58:36.842077 8539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:36.842396 8539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:36.842438 8539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:36.846683 8539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:36.846767 8539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:36.846828 8539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.114.195
I20260925 22:58:36.852293 8539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.114.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.114.195
--webserver_port=0
--tserver_master_addrs=127.20.114.254:39613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.114.195
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:36.853525 8539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:36.855001 8539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:36.865912 8545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:36.865989 8539 server_base.cc:1034] running on GCE node
W20260925 22:58:36.865857 8544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:36.865863 8547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:36.866629 8539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:36.867404 8539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:36.868605 8539 hybrid_clock.cc:648] HybridClock initialized: now 1790377116868554 us; error 65 us; skew 500 ppm
I20260925 22:58:36.871145 8539 webserver.cc:460] Webserver started at http://127.20.114.195:35873/ using document root <none> and password file <none>
I20260925 22:58:36.871833 8539 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:36.871905 8539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:36.872133 8539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:36.874508 8539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-2/data/instance:
uuid: "cb4aa5cf320b4b0095574b8b196ec540"
format_stamp: "Formatted at 2026-09-25 22:58:36 on dist-test-slave-x1lh"
server_key: "f0ed959f5731975169832ff177ed1b0b"
server_key_iv: "48f474aaaf8ca8de64867f3100c15a09"
server_key_version: "encryptionkey@0"
I20260925 22:58:36.875161 8539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance:
uuid: "cb4aa5cf320b4b0095574b8b196ec540"
format_stamp: "Formatted at 2026-09-25 22:58:36 on dist-test-slave-x1lh"
server_key: "f0ed959f5731975169832ff177ed1b0b"
server_key_iv: "48f474aaaf8ca8de64867f3100c15a09"
server_key_version: "encryptionkey@0"
I20260925 22:58:36.879626 8539 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:58:36.883247 8553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:36.884843 8539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:58:36.885109 8539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-2/wal
uuid: "cb4aa5cf320b4b0095574b8b196ec540"
format_stamp: "Formatted at 2026-09-25 22:58:36 on dist-test-slave-x1lh"
server_key: "f0ed959f5731975169832ff177ed1b0b"
server_key_iv: "48f474aaaf8ca8de64867f3100c15a09"
server_key_version: "encryptionkey@0"
I20260925 22:58:36.885281 8539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:36.918715 8539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:36.919548 8539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:36.919770 8539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:36.920433 8539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:58:36.921742 8539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:58:36.921828 8539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:36.921911 8539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:58:36.921957 8539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:36.932718 8539 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.195:45437
I20260925 22:58:36.932746 8666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.195:45437 every 8 connection(s)
I20260925 22:58:36.933840 8539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-2/data/info.pb
I20260925 22:58:36.938671 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 8539
I20260925 22:58:36.938889 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-2/wal/instance
I20260925 22:58:36.939172 20939 external_mini_cluster.cc:1428] Setting key dac7bfb57d1bbd7b43a905db5dc73121
I20260925 22:58:36.944995 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:36.936003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.195:51839 (local address 127.20.114.254:39613)
0925 22:58:36.936174 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:36.936178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:36.937457 (+ 1279us) server_negotiation.cc:407] Connection header received
0925 22:58:36.939034 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:36.939038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:36.939102 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:36.939194 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:36.940467 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.941148 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.942461 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.942851 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.943008 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:36.944349 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:36.944353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:36.944356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:36.944378 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:36.944399 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:36.944407 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:36.944489 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:36.944667 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 22:58:36.944805 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 22:58:36.946012 8669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:36.936471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:39613 (local address 127.20.114.195:51839)
0925 22:58:36.937269 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:36.937314 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:36.938735 (+ 1421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:36.939415 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:36.939423 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:36.939546 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:58:36.940296 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:36.940321 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.941284 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.941288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:36.942326 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:36.942337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.943053 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.943058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:36.943267 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:36.944034 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:58:36.944036 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:58:36.944082 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:58:36.944160 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:58:36.944505 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:58:36.944516 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:58:36.944522 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:58:36.944668 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 22:58:36.945028 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":127,"threads_started":1}
I20260925 22:58:36.947472 8667 heartbeater.cc:347] Connected to a master server at 127.20.114.254:39613
I20260925 22:58:36.947751 8667 heartbeater.cc:464] Registering TS with master...
I20260925 22:58:36.948354 8667 heartbeater.cc:511] Master 127.20.114.254:39613 requested a full tablet report, sending...
I20260925 22:58:36.949697 8217 ts_manager.cc:194] Registered new tserver with Master: cb4aa5cf320b4b0095574b8b196ec540 (127.20.114.195:45437)
I20260925 22:58:36.950331 8217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.114.195:51839
I20260925 22:58:36.954106 20939 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:58:36.960530 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:36.956175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52056 (local address 127.20.114.254:39613)
0925 22:58:36.956318 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:36.956324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:36.956520 (+ 196us) server_negotiation.cc:407] Connection header received
0925 22:58:36.956635 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:36.956637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:36.956680 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:36.956747 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:36.957259 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.957770 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.958756 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.959011 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.959096 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:36.960067 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:36.960071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:36.960073 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:36.960087 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:36.960107 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:36.960115 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:36.960182 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:36.960284 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 22:58:36.960337 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:58:36.964035 8217 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:58:36.967372 8217 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:58:36.983749 8683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:36.978172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51844 (local address 127.20.114.193:39423)
0925 22:58:36.978589 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:36.978596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:36.978790 (+ 194us) server_negotiation.cc:407] Connection header received
0925 22:58:36.978877 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:36.978886 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:36.979082 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:36.979252 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:36.979552 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.980650 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.982446 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.983013 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.983179 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:36.983409 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:36.983467 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 22:58:36.983571 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":106,"threads_started":1}
I20260925 22:58:36.983745 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:36.977753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:39423 (local address 127.0.0.1:51844)
0925 22:58:36.978616 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:36.978628 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:36.978696 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:36.979230 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:36.979236 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:36.979260 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:36.979422 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:36.979430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.980759 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.980764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:36.982282 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:36.982293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.983149 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.983153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:36.983269 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:36.983310 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 22:58:36.983423 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 22:58:36.983568 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":216,"thread_start_us":87,"threads_started":1}
I20260925 22:58:36.983804 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:36.977188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:45437 (local address 127.0.0.1:54298)
0925 22:58:36.978349 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:36.978363 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:36.978440 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:36.979306 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:36.979309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:36.979321 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:36.979441 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:36.979446 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.980669 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.980675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:36.982282 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:36.982293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.983360 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.983364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:36.983464 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:36.983512 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 22:58:36.983575 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 22:58:36.983653 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1080,"mutex_wait_us":187,"thread_start_us":97,"threads_started":1}
I20260925 22:58:36.984073 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:36.977659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54298 (local address 127.20.114.195:45437)
0925 22:58:36.978780 (+ 1121us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:36.978792 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:36.978817 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:58:36.978899 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:36.978904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:36.979126 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:36.979633 (+ 507us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:36.979746 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.980560 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.982431 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.983221 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.983571 (+ 350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:36.983744 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:36.983808 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 22:58:36.983913 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1017,"thread_start_us":113,"threads_started":1}
I20260925 22:58:36.984357 8679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:36.976933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:42261 (local address 127.0.0.1:42094)
0925 22:58:36.977573 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:36.977636 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:36.977776 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:36.978583 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:36.978589 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:36.978611 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:36.978773 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:36.978780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.980279 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.980282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:36.982027 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:36.982039 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.983472 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.983475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:36.984028 (+ 553us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:36.984052 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:58:36.984128 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:58:36.984219 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":166,"thread_start_us":101,"threads_started":1}
I20260925 22:58:36.985102 8680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:36.977123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42094 (local address 127.20.114.194:42261)
0925 22:58:36.977786 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:36.977797 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:36.977825 (+ 28us) server_negotiation.cc:407] Connection header received
0925 22:58:36.977987 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:36.977994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:36.978216 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:36.978360 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:36.978916 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.980122 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.982173 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:36.983070 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:36.983210 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:36.983419 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:36.984779 (+ 1360us) server_negotiation.cc:299] Negotiation successful
0925 22:58:36.984930 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":165,"threads_started":1}
I20260925 22:58:36.987011 8601 tablet_service.cc:1467] Processing CreateTablet for tablet bfb150c653f148c5a29bb6aaf257c29f (DEFAULT_TABLE table=test-table [id=59a25e1b0c954dd58e6ecdc6b8d3ff79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:58:36.987010 8339 tablet_service.cc:1467] Processing CreateTablet for tablet bfb150c653f148c5a29bb6aaf257c29f (DEFAULT_TABLE table=test-table [id=59a25e1b0c954dd58e6ecdc6b8d3ff79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:58:36.988045 8601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfb150c653f148c5a29bb6aaf257c29f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:36.988045 8339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfb150c653f148c5a29bb6aaf257c29f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:36.992628 8468 tablet_service.cc:1467] Processing CreateTablet for tablet bfb150c653f148c5a29bb6aaf257c29f (DEFAULT_TABLE table=test-table [id=59a25e1b0c954dd58e6ecdc6b8d3ff79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:58:36.993968 8468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfb150c653f148c5a29bb6aaf257c29f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:36.996158 8684 tablet_bootstrap.cc:492] T bfb150c653f148c5a29bb6aaf257c29f P cb4aa5cf320b4b0095574b8b196ec540: Bootstrap starting.
I20260925 22:58:36.998500 8684 tablet_bootstrap.cc:654] T bfb150c653f148c5a29bb6aaf257c29f P cb4aa5cf320b4b0095574b8b196ec540: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:36.999502 8684 log.cc:824] T bfb150c653f148c5a29bb6aaf257c29f P cb4aa5cf320b4b0095574b8b196ec540: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:37.001235 8687 tablet_bootstrap.cc:492] T bfb150c653f148c5a29bb6aaf257c29f P 58f8176383e44d48b77acf61bde3f5cd: Bootstrap starting.
I20260925 22:58:37.001286 8685 tablet_bootstrap.cc:492] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120: Bootstrap starting.
I20260925 22:58:37.002029 8684 tablet_bootstrap.cc:492] T bfb150c653f148c5a29bb6aaf257c29f P cb4aa5cf320b4b0095574b8b196ec540: No bootstrap required, opened a new log
I20260925 22:58:37.002261 8684 ts_tablet_manager.cc:1397] T bfb150c653f148c5a29bb6aaf257c29f P cb4aa5cf320b4b0095574b8b196ec540: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:58:37.003629 8685 tablet_bootstrap.cc:654] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:37.003737 8687 tablet_bootstrap.cc:654] T bfb150c653f148c5a29bb6aaf257c29f P 58f8176383e44d48b77acf61bde3f5cd: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:37.004614 8685 log.cc:824] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:37.004961 8687 log.cc:824] T bfb150c653f148c5a29bb6aaf257c29f P 58f8176383e44d48b77acf61bde3f5cd: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:37.007185 8684 raft_consensus.cc:348] T bfb150c653f148c5a29bb6aaf257c29f P cb4aa5cf320b4b0095574b8b196ec540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f8176383e44d48b77acf61bde3f5cd" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42261 } } peers { permanent_uuid: "cb4aa5cf320b4b0095574b8b196ec540" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45437 } } peers { permanent_uuid: "2b5d54dcb75c46aaa07077afb4a6c120" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39423 } }
I20260925 22:58:37.007485 8684 raft_consensus.cc:374] T bfb150c653f148c5a29bb6aaf257c29f P cb4aa5cf320b4b0095574b8b196ec540 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:37.007525 8687 tablet_bootstrap.cc:492] T bfb150c653f148c5a29bb6aaf257c29f P 58f8176383e44d48b77acf61bde3f5cd: No bootstrap required, opened a new log
I20260925 22:58:37.007525 8685 tablet_bootstrap.cc:492] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120: No bootstrap required, opened a new log
I20260925 22:58:37.007543 8684 raft_consensus.cc:729] T bfb150c653f148c5a29bb6aaf257c29f P cb4aa5cf320b4b0095574b8b196ec540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb4aa5cf320b4b0095574b8b196ec540, State: Initialized, Role: FOLLOWER
I20260925 22:58:37.007750 8687 ts_tablet_manager.cc:1397] T bfb150c653f148c5a29bb6aaf257c29f P 58f8176383e44d48b77acf61bde3f5cd: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 22:58:37.007824 8685 ts_tablet_manager.cc:1397] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:58:37.008071 8684 consensus_queue.cc:260] T bfb150c653f148c5a29bb6aaf257c29f P cb4aa5cf320b4b0095574b8b196ec540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f8176383e44d48b77acf61bde3f5cd" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42261 } } peers { permanent_uuid: "cb4aa5cf320b4b0095574b8b196ec540" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45437 } } peers { permanent_uuid: "2b5d54dcb75c46aaa07077afb4a6c120" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39423 } }
I20260925 22:58:37.008797 8667 heartbeater.cc:503] Master 127.20.114.254:39613 was elected leader, sending a full tablet report...
I20260925 22:58:37.011410 8684 ts_tablet_manager.cc:1428] T bfb150c653f148c5a29bb6aaf257c29f P cb4aa5cf320b4b0095574b8b196ec540: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260925 22:58:37.012660 8687 raft_consensus.cc:348] T bfb150c653f148c5a29bb6aaf257c29f P 58f8176383e44d48b77acf61bde3f5cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f8176383e44d48b77acf61bde3f5cd" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42261 } } peers { permanent_uuid: "cb4aa5cf320b4b0095574b8b196ec540" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45437 } } peers { permanent_uuid: "2b5d54dcb75c46aaa07077afb4a6c120" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39423 } }
I20260925 22:58:37.012660 8685 raft_consensus.cc:348] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f8176383e44d48b77acf61bde3f5cd" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42261 } } peers { permanent_uuid: "cb4aa5cf320b4b0095574b8b196ec540" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45437 } } peers { permanent_uuid: "2b5d54dcb75c46aaa07077afb4a6c120" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39423 } }
I20260925 22:58:37.013006 8687 raft_consensus.cc:374] T bfb150c653f148c5a29bb6aaf257c29f P 58f8176383e44d48b77acf61bde3f5cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:37.013006 8685 raft_consensus.cc:374] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:37.013064 8687 raft_consensus.cc:729] T bfb150c653f148c5a29bb6aaf257c29f P 58f8176383e44d48b77acf61bde3f5cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f8176383e44d48b77acf61bde3f5cd, State: Initialized, Role: FOLLOWER
I20260925 22:58:37.013064 8685 raft_consensus.cc:729] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b5d54dcb75c46aaa07077afb4a6c120, State: Initialized, Role: FOLLOWER
I20260925 22:58:37.013595 8685 consensus_queue.cc:260] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f8176383e44d48b77acf61bde3f5cd" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42261 } } peers { permanent_uuid: "cb4aa5cf320b4b0095574b8b196ec540" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45437 } } peers { permanent_uuid: "2b5d54dcb75c46aaa07077afb4a6c120" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39423 } }
I20260925 22:58:37.013595 8687 consensus_queue.cc:260] T bfb150c653f148c5a29bb6aaf257c29f P 58f8176383e44d48b77acf61bde3f5cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f8176383e44d48b77acf61bde3f5cd" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42261 } } peers { permanent_uuid: "cb4aa5cf320b4b0095574b8b196ec540" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45437 } } peers { permanent_uuid: "2b5d54dcb75c46aaa07077afb4a6c120" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39423 } }
I20260925 22:58:37.014603 8536 heartbeater.cc:503] Master 127.20.114.254:39613 was elected leader, sending a full tablet report...
I20260925 22:58:37.014762 8685 ts_tablet_manager.cc:1428] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:58:37.014762 8687 ts_tablet_manager.cc:1428] T bfb150c653f148c5a29bb6aaf257c29f P 58f8176383e44d48b77acf61bde3f5cd: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 22:58:37.015877 8405 heartbeater.cc:503] Master 127.20.114.254:39613 was elected leader, sending a full tablet report...
I20260925 22:58:37.153266 8692 raft_consensus.cc:482] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:58:37.153545 8692 raft_consensus.cc:504] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f8176383e44d48b77acf61bde3f5cd" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42261 } } peers { permanent_uuid: "cb4aa5cf320b4b0095574b8b196ec540" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45437 } } peers { permanent_uuid: "2b5d54dcb75c46aaa07077afb4a6c120" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39423 } }
I20260925 22:58:37.154870 8692 leader_election.cc:288] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58f8176383e44d48b77acf61bde3f5cd (127.20.114.194:42261), cb4aa5cf320b4b0095574b8b196ec540 (127.20.114.195:45437)
I20260925 22:58:37.161130 8680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:37.155400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:47613 (local address 127.20.114.194:42261)
0925 22:58:37.155586 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:37.155590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:37.155706 (+ 116us) server_negotiation.cc:407] Connection header received
0925 22:58:37.155907 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:37.155947 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:37.156005 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:37.156086 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:37.156686 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:37.157552 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:37.159574 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:37.160250 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:37.160394 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:37.160449 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:37.160907 (+ 458us) server_negotiation.cc:299] Negotiation successful
0925 22:58:37.160996 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 22:58:37.161130 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:37.155503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.114.193:34681 (local address 127.20.114.195:45437)
0925 22:58:37.155624 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:37.155628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:37.155707 (+ 79us) server_negotiation.cc:407] Connection header received
0925 22:58:37.155906 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:37.155910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:37.155967 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:37.156062 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:58:37.156686 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:37.157552 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:37.159570 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:37.160262 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:37.160394 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:37.160449 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:37.160907 (+ 458us) server_negotiation.cc:299] Negotiation successful
0925 22:58:37.160996 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:58:37.161151 8693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:37.155153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.194:42261 (local address 127.20.114.193:47613)
0925 22:58:37.155611 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:37.155627 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:37.155749 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:37.156127 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:37.156131 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:37.156183 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:37.156508 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:37.156518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:37.157694 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:37.157698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:37.159410 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:37.159417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:37.160434 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:37.160439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:37.160635 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:37.160700 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 22:58:37.160782 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 22:58:37.160940 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":320,"spinlock_wait_cycles":11648,"thread_start_us":132,"threads_started":1}
I20260925 22:58:37.161161 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:37.155150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.195:45437 (local address 127.20.114.193:34681)
0925 22:58:37.155581 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:37.155599 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:58:37.155772 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:58:37.156127 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:58:37.156131 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:58:37.156181 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:58:37.156519 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:37.156525 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:37.157694 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:37.157698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:37.159396 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:58:37.159412 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:58:37.160434 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:58:37.160439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:58:37.160614 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:37.160685 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 22:58:37.160782 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 22:58:37.161027 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":243,"spinlock_wait_cycles":35456,"thread_start_us":129,"threads_started":1}
I20260925 22:58:37.162151 8490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb150c653f148c5a29bb6aaf257c29f" candidate_uuid: "2b5d54dcb75c46aaa07077afb4a6c120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f8176383e44d48b77acf61bde3f5cd" is_pre_election: true
I20260925 22:58:37.162151 8621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb150c653f148c5a29bb6aaf257c29f" candidate_uuid: "2b5d54dcb75c46aaa07077afb4a6c120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb4aa5cf320b4b0095574b8b196ec540" is_pre_election: true
I20260925 22:58:37.162503 8621 raft_consensus.cc:2454] T bfb150c653f148c5a29bb6aaf257c29f P cb4aa5cf320b4b0095574b8b196ec540 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b5d54dcb75c46aaa07077afb4a6c120 in term 0.
I20260925 22:58:37.162503 8490 raft_consensus.cc:2454] T bfb150c653f148c5a29bb6aaf257c29f P 58f8176383e44d48b77acf61bde3f5cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b5d54dcb75c46aaa07077afb4a6c120 in term 0.
I20260925 22:58:37.163117 8295 leader_election.cc:302] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2b5d54dcb75c46aaa07077afb4a6c120, 58f8176383e44d48b77acf61bde3f5cd, cb4aa5cf320b4b0095574b8b196ec540; no voters:
I20260925 22:58:37.163431 8692 raft_consensus.cc:2781] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:58:37.163516 8692 raft_consensus.cc:482] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:58:37.163549 8692 raft_consensus.cc:3037] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:37.164762 8692 raft_consensus.cc:504] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f8176383e44d48b77acf61bde3f5cd" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42261 } } peers { permanent_uuid: "cb4aa5cf320b4b0095574b8b196ec540" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45437 } } peers { permanent_uuid: "2b5d54dcb75c46aaa07077afb4a6c120" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39423 } }
I20260925 22:58:37.165212 8692 leader_election.cc:288] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120 [CANDIDATE]: Term 1 election: Requested vote from peers 58f8176383e44d48b77acf61bde3f5cd (127.20.114.194:42261), cb4aa5cf320b4b0095574b8b196ec540 (127.20.114.195:45437)
I20260925 22:58:37.165750 8490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb150c653f148c5a29bb6aaf257c29f" candidate_uuid: "2b5d54dcb75c46aaa07077afb4a6c120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f8176383e44d48b77acf61bde3f5cd"
I20260925 22:58:37.165750 8621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb150c653f148c5a29bb6aaf257c29f" candidate_uuid: "2b5d54dcb75c46aaa07077afb4a6c120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb4aa5cf320b4b0095574b8b196ec540"
I20260925 22:58:37.165900 8490 raft_consensus.cc:3037] T bfb150c653f148c5a29bb6aaf257c29f P 58f8176383e44d48b77acf61bde3f5cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:37.165900 8621 raft_consensus.cc:3037] T bfb150c653f148c5a29bb6aaf257c29f P cb4aa5cf320b4b0095574b8b196ec540 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:37.167292 8621 raft_consensus.cc:2454] T bfb150c653f148c5a29bb6aaf257c29f P cb4aa5cf320b4b0095574b8b196ec540 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b5d54dcb75c46aaa07077afb4a6c120 in term 1.
I20260925 22:58:37.167292 8490 raft_consensus.cc:2454] T bfb150c653f148c5a29bb6aaf257c29f P 58f8176383e44d48b77acf61bde3f5cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b5d54dcb75c46aaa07077afb4a6c120 in term 1.
I20260925 22:58:37.167722 8294 leader_election.cc:302] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b5d54dcb75c46aaa07077afb4a6c120, 58f8176383e44d48b77acf61bde3f5cd; no voters:
I20260925 22:58:37.167971 8692 raft_consensus.cc:2781] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:37.168208 8692 raft_consensus.cc:686] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120 [term 1 LEADER]: Becoming Leader. State: Replica: 2b5d54dcb75c46aaa07077afb4a6c120, State: Running, Role: LEADER
I20260925 22:58:37.168581 8692 consensus_queue.cc:237] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f8176383e44d48b77acf61bde3f5cd" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42261 } } peers { permanent_uuid: "cb4aa5cf320b4b0095574b8b196ec540" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45437 } } peers { permanent_uuid: "2b5d54dcb75c46aaa07077afb4a6c120" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39423 } }
I20260925 22:58:37.172101 8216 catalog_manager.cc:5495] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b5d54dcb75c46aaa07077afb4a6c120 (127.20.114.193). New cstate: current_term: 1 leader_uuid: "2b5d54dcb75c46aaa07077afb4a6c120" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f8176383e44d48b77acf61bde3f5cd" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb4aa5cf320b4b0095574b8b196ec540" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b5d54dcb75c46aaa07077afb4a6c120" member_type: VOTER last_known_addr { host: "127.20.114.193" port: 39423 } health_report { overall_health: HEALTHY } } }
W20260925 22:58:37.185132 8668 tablet.cc:2369] T bfb150c653f148c5a29bb6aaf257c29f P cb4aa5cf320b4b0095574b8b196ec540: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:58:37.217206 8537 tablet.cc:2369] T bfb150c653f148c5a29bb6aaf257c29f P 58f8176383e44d48b77acf61bde3f5cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:58:37.228399 8683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:37.224657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51858 (local address 127.20.114.193:39423)
0925 22:58:37.224911 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:37.224918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:37.224934 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:58:37.225017 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:37.225019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:37.225075 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:37.225191 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:58:37.225475 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:37.225991 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:37.226895 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:37.227121 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:37.227238 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:37.227565 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:58:37.227982 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:58:37.228117 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:37.228180 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 22:58:37.228241 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 22:58:37.235471 8621 raft_consensus.cc:1261] T bfb150c653f148c5a29bb6aaf257c29f P cb4aa5cf320b4b0095574b8b196ec540 [term 1 FOLLOWER]: Refusing update from remote peer 2b5d54dcb75c46aaa07077afb4a6c120: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:58:37.235474 8490 raft_consensus.cc:1261] T bfb150c653f148c5a29bb6aaf257c29f P 58f8176383e44d48b77acf61bde3f5cd [term 1 FOLLOWER]: Refusing update from remote peer 2b5d54dcb75c46aaa07077afb4a6c120: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:58:37.236300 8692 consensus_queue.cc:1035] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f8176383e44d48b77acf61bde3f5cd" member_type: VOTER last_known_addr { host: "127.20.114.194" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:58:37.236558 8695 consensus_queue.cc:1035] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb4aa5cf320b4b0095574b8b196ec540" member_type: VOTER last_known_addr { host: "127.20.114.195" port: 45437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:58:37.254240 8216 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:58:37.254441 8216 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:58:37.257335 8216 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b657b96f7464cf6b23a3bc6b179a976: Sending DeleteTablet for 3 replicas of tablet bfb150c653f148c5a29bb6aaf257c29f
I20260925 22:58:37.258279 8339 tablet_service.cc:1514] Processing DeleteTablet for tablet bfb150c653f148c5a29bb6aaf257c29f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:58:37 UTC) from {username='slave'} at 127.0.0.1:51844
I20260925 22:58:37.258273 8468 tablet_service.cc:1514] Processing DeleteTablet for tablet bfb150c653f148c5a29bb6aaf257c29f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:58:37 UTC) from {username='slave'} at 127.0.0.1:42094
I20260925 22:58:37.259181 8710 tablet_replica.cc:330] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120: stopping tablet replica
I20260925 22:58:37.259151 8601 tablet_service.cc:1514] Processing DeleteTablet for tablet bfb150c653f148c5a29bb6aaf257c29f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:58:37 UTC) from {username='slave'} at 127.0.0.1:54298
I20260925 22:58:37.259549 8710 raft_consensus.cc:2229] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:58:37.259672 8711 tablet_replica.cc:330] T bfb150c653f148c5a29bb6aaf257c29f P cb4aa5cf320b4b0095574b8b196ec540: stopping tablet replica
I20260925 22:58:37.259934 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 8277
I20260925 22:58:37.259938 8711 raft_consensus.cc:2229] T bfb150c653f148c5a29bb6aaf257c29f P cb4aa5cf320b4b0095574b8b196ec540 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:58:37.260021 8710 raft_consensus.cc:2258] T bfb150c653f148c5a29bb6aaf257c29f P 2b5d54dcb75c46aaa07077afb4a6c120 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:58:37.260237 8711 raft_consensus.cc:2258] T bfb150c653f148c5a29bb6aaf257c29f P cb4aa5cf320b4b0095574b8b196ec540 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:58:37.260368 8709 tablet_replica.cc:330] T bfb150c653f148c5a29bb6aaf257c29f P 58f8176383e44d48b77acf61bde3f5cd: stopping tablet replica
I20260925 22:58:37.260746 8709 raft_consensus.cc:2229] T bfb150c653f148c5a29bb6aaf257c29f P 58f8176383e44d48b77acf61bde3f5cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:58:37.261070 8709 raft_consensus.cc:2258] T bfb150c653f148c5a29bb6aaf257c29f P 58f8176383e44d48b77acf61bde3f5cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:58:37.262838 8709 ts_tablet_manager.cc:1905] T bfb150c653f148c5a29bb6aaf257c29f P 58f8176383e44d48b77acf61bde3f5cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:58:37.262838 8711 ts_tablet_manager.cc:1905] T bfb150c653f148c5a29bb6aaf257c29f P cb4aa5cf320b4b0095574b8b196ec540: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:58:37.266381 8709 ts_tablet_manager.cc:1918] T bfb150c653f148c5a29bb6aaf257c29f P 58f8176383e44d48b77acf61bde3f5cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:58:37.266381 8711 ts_tablet_manager.cc:1918] T bfb150c653f148c5a29bb6aaf257c29f P cb4aa5cf320b4b0095574b8b196ec540: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:58:37.266523 8711 log.cc:1196] T bfb150c653f148c5a29bb6aaf257c29f P cb4aa5cf320b4b0095574b8b196ec540: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-2/wal/wals/bfb150c653f148c5a29bb6aaf257c29f
I20260925 22:58:37.266523 8709 log.cc:1196] T bfb150c653f148c5a29bb6aaf257c29f P 58f8176383e44d48b77acf61bde3f5cd: Deleting WAL directory at /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376856092611-20939-0/minicluster-data/ts-1/wal/wals/bfb150c653f148c5a29bb6aaf257c29f
I20260925 22:58:37.266896 8711 ts_tablet_manager.cc:1939] T bfb150c653f148c5a29bb6aaf257c29f P cb4aa5cf320b4b0095574b8b196ec540: Deleting consensus metadata
I20260925 22:58:37.266920 8709 ts_tablet_manager.cc:1939] T bfb150c653f148c5a29bb6aaf257c29f P 58f8176383e44d48b77acf61bde3f5cd: Deleting consensus metadata
W20260925 22:58:37.267982 8204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.114.193:39423 (error 108)
I20260925 22:58:37.268461 8203 catalog_manager.cc:4841] TS 58f8176383e44d48b77acf61bde3f5cd (127.20.114.194:42261): tablet bfb150c653f148c5a29bb6aaf257c29f (table test-table [id=59a25e1b0c954dd58e6ecdc6b8d3ff79]) successfully deleted
I20260925 22:58:37.268816 8204 catalog_manager.cc:4841] TS cb4aa5cf320b4b0095574b8b196ec540 (127.20.114.195:45437): tablet bfb150c653f148c5a29bb6aaf257c29f (table test-table [id=59a25e1b0c954dd58e6ecdc6b8d3ff79]) successfully deleted
I20260925 22:58:37.268966 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 8408
I20260925 22:58:37.270241 8679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:37.269748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.193:39423 (local address 127.0.0.1:51870)
0925 22:58:37.270038 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 22:58:37.270129 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.193:39423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
W20260925 22:58:37.270439 8204 catalog_manager.cc:4584] TS 2b5d54dcb75c46aaa07077afb4a6c120 (127.20.114.193:39423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bfb150c653f148c5a29bb6aaf257c29f: Network error: Client connection negotiation failed: client connection to 127.20.114.193:39423: connect: Connection refused (error 111)
I20260925 22:58:37.275758 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 8539
I20260925 22:58:37.282053 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 8186
2026-09-25T22:58:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1273 ms)
[----------] 6 tests from EncryptionPolicyTest (11014 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:58:37 dist-test-slave-x1lh krb5kdc[8715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:58:37 dist-test-slave-x1lh krb5kdc[8715](info): set up 2 sockets
Sep 25 22:58:37 dist-test-slave-x1lh krb5kdc[8715](info): commencing operation
krb5kdc: starting...
W20260925 22:58:38.345269 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:58:38 dist-test-slave-x1lh krb5kdc[8715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:58:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:58:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:58:38.481446 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44149
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:38.610620 8728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:38.611020 8728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:38.611130 8728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:38.616204 8728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:58:38.616358 8728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:38.616406 8728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:38.616451 8728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:58:38.616492 8728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:58:38.623721 8728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44149
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:38.625094 8728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:38.626276 8728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:38.637246 8733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:38.637303 8728 server_base.cc:1034] running on GCE node
W20260925 22:58:38.637351 8736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:38.637246 8734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:38.638412 8728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:38.640117 8728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:38.643559 8728 hybrid_clock.cc:648] HybridClock initialized: now 1790377118643484 us; error 68 us; skew 500 ppm
Sep 25 22:58:38 dist-test-slave-x1lh krb5kdc[8715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:38.647405 8728 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:58:38.648584 8728 webserver.cc:460] Webserver started at http://127.0.0.1:33111/ using document root <none> and password file <none>
I20260925 22:58:38.649397 8728 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:38.649516 8728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:38.649799 8728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:38.652130 8728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "d0c481ba3d424995a48d077f7ab0a4ce"
format_stamp: "Formatted at 2026-09-25 22:58:38 on dist-test-slave-x1lh"
server_key: "885eeb98fa11b9cd76f68ea457e6a1ba"
server_key_iv: "6efa010d623a1f33786c5936fe5d2c24"
server_key_version: "encryptionkey@0"
I20260925 22:58:38.652792 8728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "d0c481ba3d424995a48d077f7ab0a4ce"
format_stamp: "Formatted at 2026-09-25 22:58:38 on dist-test-slave-x1lh"
server_key: "885eeb98fa11b9cd76f68ea457e6a1ba"
server_key_iv: "6efa010d623a1f33786c5936fe5d2c24"
server_key_version: "encryptionkey@0"
I20260925 22:58:38.657312 8728 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:58:38.660461 8743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:38.662148 8728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:38.662350 8728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "d0c481ba3d424995a48d077f7ab0a4ce"
format_stamp: "Formatted at 2026-09-25 22:58:38 on dist-test-slave-x1lh"
server_key: "885eeb98fa11b9cd76f68ea457e6a1ba"
server_key_iv: "6efa010d623a1f33786c5936fe5d2c24"
server_key_version: "encryptionkey@0"
I20260925 22:58:38.662544 8728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:38.685446 8728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:38.689811 8728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:38.690085 8728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:38.698661 8728 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44149
I20260925 22:58:38.698689 8795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44149 every 8 connection(s)
I20260925 22:58:38.699792 8728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:58:38.700184 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 8728
I20260925 22:58:38.700361 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:58:38.700695 20939 external_mini_cluster.cc:1428] Setting key a274c1b2d03b93e75cdca48e7dcc8b90
I20260925 22:58:38.704061 8796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:38.712749 8796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce: Bootstrap starting.
Sep 25 22:58:38 dist-test-slave-x1lh krb5kdc[8715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:58:38.716274 8796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:38.717361 8796 log.cc:824] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:38.719722 8799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:38.702569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55791 (local address 127.0.0.1:44149)
0925 22:58:38.703099 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:38.703121 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:38.703172 (+ 51us) server_negotiation.cc:407] Connection header received
0925 22:58:38.704239 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:38.704279 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:38.704788 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:38.705248 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:38.706117 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:38.707805 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:38.709220 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:38.710056 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:38.710278 (+ 222us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:38.713693 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:38.713700 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:38.713718 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:38.713774 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:38.715985 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:38.716743 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:38.716750 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:38.716757 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:38.716959 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:38.717411 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:38.717415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:38.717417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:38.718007 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:38.718262 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:38.718649 (+ 387us) server_negotiation.cc:299] Negotiation successful
0925 22:58:38.718910 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":158,"threads_started":1}
I20260925 22:58:38.720108 8796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce: No bootstrap required, opened a new log
I20260925 22:58:38.723567 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:58:38.725344 8796 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c481ba3d424995a48d077f7ab0a4ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } }
I20260925 22:58:38.725718 8796 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:38.725806 8796 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0c481ba3d424995a48d077f7ab0a4ce, State: Initialized, Role: FOLLOWER
I20260925 22:58:38.726365 8796 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c481ba3d424995a48d077f7ab0a4ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } }
I20260925 22:58:38.726513 8796 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:58:38.726609 8796 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:58:38.726722 8796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:38.727957 8796 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c481ba3d424995a48d077f7ab0a4ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } }
I20260925 22:58:38.728397 8796 leader_election.cc:302] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0c481ba3d424995a48d077f7ab0a4ce; no voters:
I20260925 22:58:38.728842 8796 leader_election.cc:288] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:58:38.728986 8806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:38.729228 8806 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce [term 1 LEADER]: Becoming Leader. State: Replica: d0c481ba3d424995a48d077f7ab0a4ce, State: Running, Role: LEADER
I20260925 22:58:38.729614 8806 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c481ba3d424995a48d077f7ab0a4ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } }
I20260925 22:58:38.730459 8796 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:58:38.731125 8806 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0c481ba3d424995a48d077f7ab0a4ce. Latest consensus state: current_term: 1 leader_uuid: "d0c481ba3d424995a48d077f7ab0a4ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c481ba3d424995a48d077f7ab0a4ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } } }
I20260925 22:58:38.731261 8806 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:38.731606 8807 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0c481ba3d424995a48d077f7ab0a4ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c481ba3d424995a48d077f7ab0a4ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } } }
I20260925 22:58:38.731730 8807 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce [sys.catalog]: This master's current role is: LEADER
W20260925 22:58:38.736898 8821 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:58:38.737036 8821 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:58:38.737106 8810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:58:38.737308 8799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:38.726094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49829 (local address 127.0.0.1:44149)
0925 22:58:38.726247 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:38.726252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:38.726569 (+ 317us) server_negotiation.cc:407] Connection header received
0925 22:58:38.726845 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:38.726848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:38.726916 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:38.727010 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:38.727820 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:38.728604 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:38.729949 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:38.730303 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:38.730386 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:38.732939 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:38.732946 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:38.732949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:38.732984 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:38.734867 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:38.735593 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:38.735597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:38.735599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:38.735694 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:38.736085 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:38.736088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:38.736090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:38.736289 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:38.736452 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:38.736745 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 22:58:38.736896 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 22:58:38.738104 8810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:58:38.744797 8810 catalog_manager.cc:1347] Generated new cluster ID: 5d38b94450b04e13901b20b65635e097
I20260925 22:58:38.744941 8810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:58:38.755765 8810 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:58:38.757433 8810 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:58:38.778872 8810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0c481ba3d424995a48d077f7ab0a4ce: Generated new TSK 0
I20260925 22:58:38.779762 8810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:58:38.790614 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 8728
W20260925 22:58:38.796800 8713 connection.cc:535] client connection to 127.0.0.1:44149 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44149): Connection reset by peer (error 104)
2026-09-25T22:58:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1507 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:58:38 dist-test-slave-x1lh krb5kdc[8827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:58:38 dist-test-slave-x1lh krb5kdc[8827](info): set up 2 sockets
Sep 25 22:58:38 dist-test-slave-x1lh krb5kdc[8827](info): commencing operation
krb5kdc: starting...
W20260925 22:58:39.921903 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:58:40 dist-test-slave-x1lh krb5kdc[8827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:58:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:58:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:58:40.074862 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38235
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:40.202129 8840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:40.202522 8840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:40.202618 8840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:40.207399 8840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:58:40.207563 8840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:40.207603 8840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:40.207649 8840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:58:40.207705 8840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:58:40.213933 8840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38235
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:40.215245 8840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:40.216385 8840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:40.227717 8845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:40.227717 8848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:40.227843 8846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:40.229517 8840 server_base.cc:1034] running on GCE node
I20260925 22:58:40.230309 8840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:40.232111 8840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:40.233515 8840 hybrid_clock.cc:648] HybridClock initialized: now 1790377120233479 us; error 67 us; skew 500 ppm
Sep 25 22:58:40 dist-test-slave-x1lh krb5kdc[8827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:40.243963 8840 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:58:40.245319 8840 webserver.cc:460] Webserver started at http://127.0.0.1:42633/ using document root <none> and password file <none>
I20260925 22:58:40.245988 8840 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:40.246099 8840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:40.246367 8840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:40.249265 8840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "f793adc9db964567892dcc3aab8bdabe"
format_stamp: "Formatted at 2026-09-25 22:58:40 on dist-test-slave-x1lh"
server_key: "12c59fc28c2bee16195c95db1a5ff0f7"
server_key_iv: "b304d0ebd1b38145eb6cbb86cc881b4b"
server_key_version: "encryptionkey@0"
I20260925 22:58:40.250223 8840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "f793adc9db964567892dcc3aab8bdabe"
format_stamp: "Formatted at 2026-09-25 22:58:40 on dist-test-slave-x1lh"
server_key: "12c59fc28c2bee16195c95db1a5ff0f7"
server_key_iv: "b304d0ebd1b38145eb6cbb86cc881b4b"
server_key_version: "encryptionkey@0"
I20260925 22:58:40.255326 8840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 22:58:40.258862 8855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:40.260715 8840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:58:40.261029 8840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "f793adc9db964567892dcc3aab8bdabe"
format_stamp: "Formatted at 2026-09-25 22:58:40 on dist-test-slave-x1lh"
server_key: "12c59fc28c2bee16195c95db1a5ff0f7"
server_key_iv: "b304d0ebd1b38145eb6cbb86cc881b4b"
server_key_version: "encryptionkey@0"
I20260925 22:58:40.261199 8840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:40.284964 8840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:40.289090 8840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:40.289350 8840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:40.298676 8840 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38235
I20260925 22:58:40.298722 8907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38235 every 8 connection(s)
I20260925 22:58:40.299919 8840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:58:40.303227 8908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:40.303928 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 8840
I20260925 22:58:40.304092 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:58:40.304437 20939 external_mini_cluster.cc:1428] Setting key 38efb5e8a601c43c3376bff13075dadd
I20260925 22:58:40.312078 8908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f793adc9db964567892dcc3aab8bdabe: Bootstrap starting.
I20260925 22:58:40.315649 8908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f793adc9db964567892dcc3aab8bdabe: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:40.316951 8911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:40.306251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57247 (local address 127.0.0.1:38235)
0925 22:58:40.306816 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:40.306838 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:40.306890 (+ 52us) server_negotiation.cc:407] Connection header received
0925 22:58:40.307817 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:40.307844 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:40.308309 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:40.309073 (+ 764us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:40.309615 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:40.310712 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:40.311708 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:40.312380 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:40.312588 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:40.314208 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:40.314214 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:40.314287 (+ 73us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:40.314948 (+ 661us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:40.315093 (+ 145us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:40.315132 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:40.315315 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:40.315566 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 22:58:40.315790 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":168,"threads_started":1}
I20260925 22:58:40.317215 8908 log.cc:824] T 00000000000000000000000000000000 P f793adc9db964567892dcc3aab8bdabe: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:40.320591 8908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f793adc9db964567892dcc3aab8bdabe: No bootstrap required, opened a new log
I20260925 22:58:40.323112 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:58:40.325181 8908 raft_consensus.cc:348] T 00000000000000000000000000000000 P f793adc9db964567892dcc3aab8bdabe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f793adc9db964567892dcc3aab8bdabe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } }
I20260925 22:58:40.325528 8908 raft_consensus.cc:374] T 00000000000000000000000000000000 P f793adc9db964567892dcc3aab8bdabe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:40.325698 8908 raft_consensus.cc:729] T 00000000000000000000000000000000 P f793adc9db964567892dcc3aab8bdabe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f793adc9db964567892dcc3aab8bdabe, State: Initialized, Role: FOLLOWER
I20260925 22:58:40.326336 8908 consensus_queue.cc:260] T 00000000000000000000000000000000 P f793adc9db964567892dcc3aab8bdabe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f793adc9db964567892dcc3aab8bdabe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } }
I20260925 22:58:40.326538 8908 raft_consensus.cc:388] T 00000000000000000000000000000000 P f793adc9db964567892dcc3aab8bdabe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:58:40.326617 8908 raft_consensus.cc:482] T 00000000000000000000000000000000 P f793adc9db964567892dcc3aab8bdabe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:58:40.326740 8908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f793adc9db964567892dcc3aab8bdabe [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:40.327951 8908 raft_consensus.cc:504] T 00000000000000000000000000000000 P f793adc9db964567892dcc3aab8bdabe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f793adc9db964567892dcc3aab8bdabe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } }
I20260925 22:58:40.328447 8908 leader_election.cc:302] T 00000000000000000000000000000000 P f793adc9db964567892dcc3aab8bdabe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f793adc9db964567892dcc3aab8bdabe; no voters:
I20260925 22:58:40.328845 8908 leader_election.cc:288] T 00000000000000000000000000000000 P f793adc9db964567892dcc3aab8bdabe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:58:40.328985 8918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f793adc9db964567892dcc3aab8bdabe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:40.329281 8918 raft_consensus.cc:686] T 00000000000000000000000000000000 P f793adc9db964567892dcc3aab8bdabe [term 1 LEADER]: Becoming Leader. State: Replica: f793adc9db964567892dcc3aab8bdabe, State: Running, Role: LEADER
I20260925 22:58:40.329738 8918 consensus_queue.cc:237] T 00000000000000000000000000000000 P f793adc9db964567892dcc3aab8bdabe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f793adc9db964567892dcc3aab8bdabe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } }
I20260925 22:58:40.330370 8911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:40.325497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39047 (local address 127.0.0.1:38235)
0925 22:58:40.325677 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:40.325682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:40.325901 (+ 219us) server_negotiation.cc:407] Connection header received
0925 22:58:40.326061 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:40.326064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:40.326123 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:40.326212 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:40.326501 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:40.327325 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:40.328271 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:40.328649 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:40.328772 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:40.329811 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:40.329815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:40.329817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:40.329838 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:40.329871 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:40.329886 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:40.330032 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:40.330172 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 22:58:40.330218 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 22:58:40.330619 8908 sys_catalog.cc:564] T 00000000000000000000000000000000 P f793adc9db964567892dcc3aab8bdabe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:58:40.332137 8919 sys_catalog.cc:455] T 00000000000000000000000000000000 P f793adc9db964567892dcc3aab8bdabe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f793adc9db964567892dcc3aab8bdabe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f793adc9db964567892dcc3aab8bdabe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } } }
I20260925 22:58:40.332293 8919 sys_catalog.cc:458] T 00000000000000000000000000000000 P f793adc9db964567892dcc3aab8bdabe [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:40.332665 8929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:58:40.332667 8920 sys_catalog.cc:455] T 00000000000000000000000000000000 P f793adc9db964567892dcc3aab8bdabe [sys.catalog]: SysCatalogTable state changed. Reason: New leader f793adc9db964567892dcc3aab8bdabe. Latest consensus state: current_term: 1 leader_uuid: "f793adc9db964567892dcc3aab8bdabe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f793adc9db964567892dcc3aab8bdabe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } } }
I20260925 22:58:40.332847 8920 sys_catalog.cc:458] T 00000000000000000000000000000000 P f793adc9db964567892dcc3aab8bdabe [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:40.336536 8929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:58:40.343712 8929 catalog_manager.cc:1347] Generated new cluster ID: 4ff72b9726f54fa5afb70c75f714a851
I20260925 22:58:40.343866 8929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:58:40.376178 8929 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:58:40.377478 8929 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:58:40.387931 8929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f793adc9db964567892dcc3aab8bdabe: Generated new TSK 0
I20260925 22:58:40.388798 8929 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:58:40.399667 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 8840
W20260925 22:58:40.405786 8825 connection.cc:535] client connection to 127.0.0.1:38235 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38235): Connection reset by peer (error 104)
2026-09-25T22:58:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1608 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:58:40 dist-test-slave-x1lh krb5kdc[8939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:58:40 dist-test-slave-x1lh krb5kdc[8939](info): set up 2 sockets
Sep 25 22:58:40 dist-test-slave-x1lh krb5kdc[8939](info): commencing operation
krb5kdc: starting...
W20260925 22:58:41.493849 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:58:41 dist-test-slave-x1lh krb5kdc[8939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:58:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:58:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:58:41.628810 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41743
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:41.746834 8952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:41.747141 8952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:41.747184 8952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:41.751554 8952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:58:41.751644 8952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:41.751665 8952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:41.751682 8952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:58:41.751698 8952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:58:41.757627 8952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41743
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:41.758828 8952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:41.759914 8952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:41.769151 8957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:41.769167 8958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:41.769284 8952 server_base.cc:1034] running on GCE node
W20260925 22:58:41.769151 8960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:41.770128 8952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:41.771364 8952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:41.772567 8952 hybrid_clock.cc:648] HybridClock initialized: now 1790377121772532 us; error 48 us; skew 500 ppm
Sep 25 22:58:41 dist-test-slave-x1lh krb5kdc[8939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:41.775661 8952 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:58:41.777033 8952 webserver.cc:460] Webserver started at http://127.0.0.1:41611/ using document root <none> and password file <none>
I20260925 22:58:41.777665 8952 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:41.777765 8952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:41.778023 8952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:41.780277 8952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "d9878b6c7bdd4bf69acfeeab7437af5c"
format_stamp: "Formatted at 2026-09-25 22:58:41 on dist-test-slave-x1lh"
server_key: "8012bfaf7e9afdc4bccd651f60534d8a"
server_key_iv: "a345a9edc1d86ce7538bbbd87d77ebe3"
server_key_version: "encryptionkey@0"
I20260925 22:58:41.781013 8952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "d9878b6c7bdd4bf69acfeeab7437af5c"
format_stamp: "Formatted at 2026-09-25 22:58:41 on dist-test-slave-x1lh"
server_key: "8012bfaf7e9afdc4bccd651f60534d8a"
server_key_iv: "a345a9edc1d86ce7538bbbd87d77ebe3"
server_key_version: "encryptionkey@0"
I20260925 22:58:41.785812 8952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:58:41.788597 8967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:41.790199 8952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:41.790382 8952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "d9878b6c7bdd4bf69acfeeab7437af5c"
format_stamp: "Formatted at 2026-09-25 22:58:41 on dist-test-slave-x1lh"
server_key: "8012bfaf7e9afdc4bccd651f60534d8a"
server_key_iv: "a345a9edc1d86ce7538bbbd87d77ebe3"
server_key_version: "encryptionkey@0"
I20260925 22:58:41.790524 8952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:41.796062 8952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:41.799660 8952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:41.799890 8952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:41.807751 8952 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41743
I20260925 22:58:41.807758 9019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41743 every 8 connection(s)
I20260925 22:58:41.808799 8952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:58:41.811909 9020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:41.816133 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 8952
I20260925 22:58:41.816319 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:58:41.816637 20939 external_mini_cluster.cc:1428] Setting key aa38958554b0d7ee96e74f354a7967a0
I20260925 22:58:41.818446 9020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9878b6c7bdd4bf69acfeeab7437af5c: Bootstrap starting.
I20260925 22:58:41.821445 9020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9878b6c7bdd4bf69acfeeab7437af5c: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:41.822515 9020 log.cc:824] T 00000000000000000000000000000000 P d9878b6c7bdd4bf69acfeeab7437af5c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:41.823241 9023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:41.818458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39709 (local address 127.0.0.1:41743)
0925 22:58:41.819026 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:41.819044 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:41.819088 (+ 44us) server_negotiation.cc:407] Connection header received
0925 22:58:41.820039 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:41.820063 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:41.820443 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:41.820838 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:41.821782 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:41.821790 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:41.821810 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:41.821824 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:41.821902 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:41.821929 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:41.822041 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:41.822142 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 22:58:41.822308 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":176,"threads_started":1}
I20260925 22:58:41.825137 9020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9878b6c7bdd4bf69acfeeab7437af5c: No bootstrap required, opened a new log
I20260925 22:58:41.826622 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:58:41.829864 9020 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9878b6c7bdd4bf69acfeeab7437af5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9878b6c7bdd4bf69acfeeab7437af5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41743 } }
I20260925 22:58:41.830229 9020 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9878b6c7bdd4bf69acfeeab7437af5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:41.830289 9020 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9878b6c7bdd4bf69acfeeab7437af5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9878b6c7bdd4bf69acfeeab7437af5c, State: Initialized, Role: FOLLOWER
I20260925 22:58:41.830825 9020 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9878b6c7bdd4bf69acfeeab7437af5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9878b6c7bdd4bf69acfeeab7437af5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41743 } }
I20260925 22:58:41.830916 9023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:41.828890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57569 (local address 127.0.0.1:41743)
0925 22:58:41.829022 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:41.829027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:41.829174 (+ 147us) server_negotiation.cc:407] Connection header received
0925 22:58:41.829266 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:41.829269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:41.829317 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:41.829406 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:41.830230 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:41.830233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:41.830237 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:41.830252 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:41.830291 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:41.830309 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:41.830402 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:41.830619 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 22:58:41.830674 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:58:41.830971 9020 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9878b6c7bdd4bf69acfeeab7437af5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:58:41.831048 9020 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9878b6c7bdd4bf69acfeeab7437af5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:58:41.831149 9020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9878b6c7bdd4bf69acfeeab7437af5c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:41.832363 9020 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9878b6c7bdd4bf69acfeeab7437af5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9878b6c7bdd4bf69acfeeab7437af5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41743 } }
I20260925 22:58:41.832747 9020 leader_election.cc:302] T 00000000000000000000000000000000 P d9878b6c7bdd4bf69acfeeab7437af5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9878b6c7bdd4bf69acfeeab7437af5c; no voters:
I20260925 22:58:41.833150 9020 leader_election.cc:288] T 00000000000000000000000000000000 P d9878b6c7bdd4bf69acfeeab7437af5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:58:41.833307 9030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9878b6c7bdd4bf69acfeeab7437af5c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:41.833587 9030 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9878b6c7bdd4bf69acfeeab7437af5c [term 1 LEADER]: Becoming Leader. State: Replica: d9878b6c7bdd4bf69acfeeab7437af5c, State: Running, Role: LEADER
I20260925 22:58:41.834014 9030 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9878b6c7bdd4bf69acfeeab7437af5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9878b6c7bdd4bf69acfeeab7437af5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41743 } }
I20260925 22:58:41.834540 9020 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9878b6c7bdd4bf69acfeeab7437af5c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:58:41.835958 9032 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9878b6c7bdd4bf69acfeeab7437af5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9878b6c7bdd4bf69acfeeab7437af5c. Latest consensus state: current_term: 1 leader_uuid: "d9878b6c7bdd4bf69acfeeab7437af5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9878b6c7bdd4bf69acfeeab7437af5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41743 } } }
I20260925 22:58:41.835996 9031 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9878b6c7bdd4bf69acfeeab7437af5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9878b6c7bdd4bf69acfeeab7437af5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9878b6c7bdd4bf69acfeeab7437af5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41743 } } }
I20260925 22:58:41.836087 9032 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9878b6c7bdd4bf69acfeeab7437af5c [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:41.836102 9031 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9878b6c7bdd4bf69acfeeab7437af5c [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:41.836517 9039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:58:41.840360 9039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:58:41.847570 9039 catalog_manager.cc:1347] Generated new cluster ID: 57bb5dfc012548908fca560305158c5d
I20260925 22:58:41.847680 9039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:58:41.865756 9039 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:58:41.866146 9039 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:58:41.872084 9039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9878b6c7bdd4bf69acfeeab7437af5c: Generated new TSK 0
I20260925 22:58:41.873107 9039 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:58:41.886690 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 8952
2026-09-25T22:58:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1486 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:58:41 dist-test-slave-x1lh krb5kdc[9051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:58:41 dist-test-slave-x1lh krb5kdc[9051](info): set up 2 sockets
Sep 25 22:58:41 dist-test-slave-x1lh krb5kdc[9051](info): commencing operation
krb5kdc: starting...
W20260925 22:58:42.941367 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:58:43 dist-test-slave-x1lh krb5kdc[9051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:58:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:58:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:58:43.078126 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34723
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:43.197886 9064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:43.198244 9064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:43.198328 9064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:43.202955 9064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:58:43.203094 9064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:43.203137 9064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:43.203173 9064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:58:43.203212 9064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:58:43.209256 9064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:43.210541 9064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:43.211663 9064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:43.221149 9069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:43.221237 9064 server_base.cc:1034] running on GCE node
W20260925 22:58:43.221149 9070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:43.221180 9072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:43.221982 9064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:43.223227 9064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:43.224444 9064 hybrid_clock.cc:648] HybridClock initialized: now 1790377123224423 us; error 38 us; skew 500 ppm
Sep 25 22:58:43 dist-test-slave-x1lh krb5kdc[9051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:43.227504 9064 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:58:43.228540 9064 webserver.cc:460] Webserver started at http://127.0.0.1:36959/ using document root <none> and password file <none>
I20260925 22:58:43.229197 9064 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:43.229269 9064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:43.229454 9064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:43.231691 9064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "a699f65d74f541af8368fe8456ce6e3a"
format_stamp: "Formatted at 2026-09-25 22:58:43 on dist-test-slave-x1lh"
server_key: "8f2ccab3eff3597a86dcfdf1e2cd477e"
server_key_iv: "4638e176db884a85d81fe4f2888c183a"
server_key_version: "encryptionkey@0"
I20260925 22:58:43.232275 9064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "a699f65d74f541af8368fe8456ce6e3a"
format_stamp: "Formatted at 2026-09-25 22:58:43 on dist-test-slave-x1lh"
server_key: "8f2ccab3eff3597a86dcfdf1e2cd477e"
server_key_iv: "4638e176db884a85d81fe4f2888c183a"
server_key_version: "encryptionkey@0"
I20260925 22:58:43.236405 9064 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:58:43.239130 9079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:43.240545 9064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:58:43.240696 9064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "a699f65d74f541af8368fe8456ce6e3a"
format_stamp: "Formatted at 2026-09-25 22:58:43 on dist-test-slave-x1lh"
server_key: "8f2ccab3eff3597a86dcfdf1e2cd477e"
server_key_iv: "4638e176db884a85d81fe4f2888c183a"
server_key_version: "encryptionkey@0"
I20260925 22:58:43.240824 9064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:43.259804 9064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:43.263481 9064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:43.263734 9064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:43.271847 9064 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34723
I20260925 22:58:43.272130 9131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34723 every 8 connection(s)
I20260925 22:58:43.273072 9064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:58:43.275683 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 9064
I20260925 22:58:43.275808 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:58:43.276127 20939 external_mini_cluster.cc:1428] Setting key a506e099c5d97350acf6d7dbc8e76d54
I20260925 22:58:43.276314 9132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:43.283887 9132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a: Bootstrap starting.
Sep 25 22:58:43 dist-test-slave-x1lh krb5kdc[9051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:58:43.286949 9132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:43.288074 9132 log.cc:824] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:43.290907 9132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a: No bootstrap required, opened a new log
I20260925 22:58:43.292493 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:43.277696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56213 (local address 127.0.0.1:34723)
0925 22:58:43.278230 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:43.278247 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:43.278287 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:58:43.279325 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:43.279357 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:43.279748 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:43.280138 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:43.281040 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:43.282079 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:43.282998 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:43.283503 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:43.283674 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:43.286516 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:43.286523 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:43.286544 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:43.286588 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:43.288958 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:43.289488 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:43.289495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:43.289502 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:43.289656 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:43.290047 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:43.290051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:43.290052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:43.290671 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:43.290897 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:43.291334 (+ 437us) server_negotiation.cc:299] Negotiation successful
0925 22:58:43.291642 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":161,"threads_started":1}
I20260925 22:58:43.295809 9132 raft_consensus.cc:348] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a699f65d74f541af8368fe8456ce6e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34723 } }
I20260925 22:58:43.296168 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:58:43.296190 9132 raft_consensus.cc:374] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:43.296281 9132 raft_consensus.cc:729] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a699f65d74f541af8368fe8456ce6e3a, State: Initialized, Role: FOLLOWER
I20260925 22:58:43.296772 9132 consensus_queue.cc:260] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a699f65d74f541af8368fe8456ce6e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34723 } }
I20260925 22:58:43.297024 9132 raft_consensus.cc:388] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:58:43.297120 9132 raft_consensus.cc:482] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:58:43.297235 9132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:43.298805 9132 raft_consensus.cc:504] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a699f65d74f541af8368fe8456ce6e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34723 } }
I20260925 22:58:43.299273 9132 leader_election.cc:302] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a699f65d74f541af8368fe8456ce6e3a; no voters:
I20260925 22:58:43.299664 9132 leader_election.cc:288] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:58:43.299769 9142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:43.300062 9142 raft_consensus.cc:686] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a [term 1 LEADER]: Becoming Leader. State: Replica: a699f65d74f541af8368fe8456ce6e3a, State: Running, Role: LEADER
I20260925 22:58:43.300488 9142 consensus_queue.cc:237] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a699f65d74f541af8368fe8456ce6e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34723 } }
I20260925 22:58:43.301151 9132 sys_catalog.cc:564] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:58:43.302318 9143 sys_catalog.cc:455] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a699f65d74f541af8368fe8456ce6e3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a699f65d74f541af8368fe8456ce6e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34723 } } }
I20260925 22:58:43.302467 9143 sys_catalog.cc:458] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:43.302774 9144 sys_catalog.cc:455] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a699f65d74f541af8368fe8456ce6e3a. Latest consensus state: current_term: 1 leader_uuid: "a699f65d74f541af8368fe8456ce6e3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a699f65d74f541af8368fe8456ce6e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34723 } } }
I20260925 22:58:43.302943 9144 sys_catalog.cc:458] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a [sys.catalog]: This master's current role is: LEADER
W20260925 22:58:43.307241 9157 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:58:43.307399 9157 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:58:43.307539 9151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:58:43.308436 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:43.298617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37953 (local address 127.0.0.1:34723)
0925 22:58:43.298752 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:43.298755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:43.299003 (+ 248us) server_negotiation.cc:407] Connection header received
0925 22:58:43.299152 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:43.299155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:43.299208 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:43.299296 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:43.300257 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:43.301072 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:43.302345 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:43.302698 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:43.302828 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:43.304679 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:43.304682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:43.304685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:43.304712 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:43.306436 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:43.307040 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:43.307043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:43.307045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:43.307140 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:43.307467 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:43.307471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:43.307472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:43.307723 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:43.307875 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:43.308133 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 22:58:43.308261 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:58:43.308542 9151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:58:43.315500 9151 catalog_manager.cc:1347] Generated new cluster ID: e6552e468858473186f2954bc36342c9
I20260925 22:58:43.315594 9151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:58:43.345603 9151 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:58:43.346657 9151 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:58:43.370398 9151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a699f65d74f541af8368fe8456ce6e3a: Generated new TSK 0
I20260925 22:58:43.371500 9151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:58:43.378928 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 9064
2026-09-25T22:58:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1492 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:58:43 dist-test-slave-x1lh krb5kdc[9163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:58:43 dist-test-slave-x1lh krb5kdc[9163](info): set up 2 sockets
Sep 25 22:58:43 dist-test-slave-x1lh krb5kdc[9163](info): commencing operation
krb5kdc: starting...
W20260925 22:58:44.457360 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:58:44 dist-test-slave-x1lh krb5kdc[9163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:58:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:58:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:58:44.609537 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40657
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:44.731266 9176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:44.731586 9176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:44.731631 9176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:44.735927 9176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:58:44.736011 9176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:44.736030 9176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:44.736048 9176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:58:44.736063 9176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:58:44.742486 9176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:44.743693 9176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:44.744764 9176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:44.754323 9184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:44.754318 9181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:44.754366 9176 server_base.cc:1034] running on GCE node
W20260925 22:58:44.754321 9182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:44.755270 9176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:44.756599 9176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:44.757812 9176 hybrid_clock.cc:648] HybridClock initialized: now 1790377124757803 us; error 73 us; skew 500 ppm
Sep 25 22:58:44 dist-test-slave-x1lh krb5kdc[9163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:44.761148 9176 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:58:44.762169 9176 webserver.cc:460] Webserver started at http://127.0.0.1:40673/ using document root <none> and password file <none>
I20260925 22:58:44.762750 9176 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:44.762817 9176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:44.762993 9176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:44.765290 9176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "a416be5710794fe4aef73d086f608ddb"
format_stamp: "Formatted at 2026-09-25 22:58:44 on dist-test-slave-x1lh"
server_key: "ea216bf6e14e2c688523549a56924cdb"
server_key_iv: "20127cf173e4ff804f57898edfbde4a7"
server_key_version: "encryptionkey@0"
I20260925 22:58:44.765863 9176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "a416be5710794fe4aef73d086f608ddb"
format_stamp: "Formatted at 2026-09-25 22:58:44 on dist-test-slave-x1lh"
server_key: "ea216bf6e14e2c688523549a56924cdb"
server_key_iv: "20127cf173e4ff804f57898edfbde4a7"
server_key_version: "encryptionkey@0"
I20260925 22:58:44.770114 9176 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:58:44.772830 9191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:44.774477 9176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:44.774676 9176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "a416be5710794fe4aef73d086f608ddb"
format_stamp: "Formatted at 2026-09-25 22:58:44 on dist-test-slave-x1lh"
server_key: "ea216bf6e14e2c688523549a56924cdb"
server_key_iv: "20127cf173e4ff804f57898edfbde4a7"
server_key_version: "encryptionkey@0"
I20260925 22:58:44.774809 9176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:44.799238 9176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:44.803660 9176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:44.803864 9176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:44.813642 9176 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40657
I20260925 22:58:44.813660 9243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40657 every 8 connection(s)
I20260925 22:58:44.815130 9176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:58:44.817456 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 9176
I20260925 22:58:44.817579 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:58:44.817905 20939 external_mini_cluster.cc:1428] Setting key c00b41dccb640642af097eb07cb866f1
I20260925 22:58:44.818655 9244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:44.825861 9244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb: Bootstrap starting.
I20260925 22:58:44.828179 9247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:44.819396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38819 (local address 127.0.0.1:40657)
0925 22:58:44.819892 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:44.819916 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:44.819956 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:58:44.820618 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:44.820635 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:44.821076 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:44.821381 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:44.821887 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:44.822927 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:44.824196 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:44.824712 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:44.824843 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:44.825885 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:44.825891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:44.825915 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:44.826500 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:44.826566 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:44.826593 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:44.826692 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:44.826896 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 22:58:44.827190 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":179,"threads_started":1}
I20260925 22:58:44.828634 9244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:44.829581 9244 log.cc:824] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:44.832023 9244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb: No bootstrap required, opened a new log
I20260925 22:58:44.832101 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:58:44.836628 9244 raft_consensus.cc:348] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a416be5710794fe4aef73d086f608ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40657 } }
I20260925 22:58:44.836969 9244 raft_consensus.cc:374] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:44.837056 9244 raft_consensus.cc:729] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a416be5710794fe4aef73d086f608ddb, State: Initialized, Role: FOLLOWER
I20260925 22:58:44.837605 9244 consensus_queue.cc:260] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a416be5710794fe4aef73d086f608ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40657 } }
I20260925 22:58:44.837782 9244 raft_consensus.cc:388] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:58:44.837858 9244 raft_consensus.cc:482] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:58:44.837966 9244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:44.839253 9247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:44.834397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55259 (local address 127.0.0.1:40657)
0925 22:58:44.834538 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:44.834545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:44.834716 (+ 171us) server_negotiation.cc:407] Connection header received
0925 22:58:44.834819 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:44.834823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:44.834888 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:44.835011 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:44.835313 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:44.836149 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:44.837231 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:44.837565 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:44.837676 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:44.838691 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:44.838695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:44.838698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:44.838713 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:44.838756 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:44.838771 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:44.838865 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:44.839009 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 22:58:44.839081 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:58:44.839205 9244 raft_consensus.cc:504] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a416be5710794fe4aef73d086f608ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40657 } }
I20260925 22:58:44.839609 9244 leader_election.cc:302] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a416be5710794fe4aef73d086f608ddb; no voters:
I20260925 22:58:44.840008 9244 leader_election.cc:288] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:58:44.840121 9254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:44.840404 9254 raft_consensus.cc:686] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb [term 1 LEADER]: Becoming Leader. State: Replica: a416be5710794fe4aef73d086f608ddb, State: Running, Role: LEADER
I20260925 22:58:44.840817 9254 consensus_queue.cc:237] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a416be5710794fe4aef73d086f608ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40657 } }
I20260925 22:58:44.841454 9244 sys_catalog.cc:564] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:58:44.842779 9256 sys_catalog.cc:455] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a416be5710794fe4aef73d086f608ddb. Latest consensus state: current_term: 1 leader_uuid: "a416be5710794fe4aef73d086f608ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a416be5710794fe4aef73d086f608ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40657 } } }
I20260925 22:58:44.842911 9256 sys_catalog.cc:458] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:44.843174 9255 sys_catalog.cc:455] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a416be5710794fe4aef73d086f608ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a416be5710794fe4aef73d086f608ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40657 } } }
I20260925 22:58:44.843272 9255 sys_catalog.cc:458] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb [sys.catalog]: This master's current role is: LEADER
W20260925 22:58:44.846977 9269 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:58:44.847100 9269 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:58:44.847169 9263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:58:44.847884 9263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:58:44.854209 9263 catalog_manager.cc:1347] Generated new cluster ID: 6c8bcf00a8464d8b902eac5b52f6b5ec
I20260925 22:58:44.854312 9263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:58:44.861268 9263 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:58:44.862632 9263 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:58:44.870545 9263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a416be5710794fe4aef73d086f608ddb: Generated new TSK 0
I20260925 22:58:44.871570 9263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:58:44.876690 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 9176
2026-09-25T22:58:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1497 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:58:44 dist-test-slave-x1lh krb5kdc[9275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:58:44 dist-test-slave-x1lh krb5kdc[9275](info): set up 2 sockets
Sep 25 22:58:44 dist-test-slave-x1lh krb5kdc[9275](info): commencing operation
krb5kdc: starting...
W20260925 22:58:45.985334 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:58:46 dist-test-slave-x1lh krb5kdc[9275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:58:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:58:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:58:46.125087 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45209
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:46.244208 9288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:46.244580 9288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:46.244658 9288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:46.249253 9288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:58:46.249356 9288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:46.249392 9288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:46.249444 9288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:58:46.249493 9288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:58:46.256253 9288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:46.257809 9288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:46.259191 9288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:46.270035 9294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:46.270193 9296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:46.270193 9293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:46.270469 9288 server_base.cc:1034] running on GCE node
I20260925 22:58:46.271195 9288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:46.272588 9288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:46.273860 9288 hybrid_clock.cc:648] HybridClock initialized: now 1790377126273841 us; error 39 us; skew 500 ppm
Sep 25 22:58:46 dist-test-slave-x1lh krb5kdc[9275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:46.283262 9288 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:58:46.284381 9288 webserver.cc:460] Webserver started at http://127.0.0.1:46183/ using document root <none> and password file <none>
I20260925 22:58:46.285099 9288 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:46.285184 9288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:46.285431 9288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:46.289390 9288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "5c1feb5081ef4a14a40b176c34dff6df"
format_stamp: "Formatted at 2026-09-25 22:58:46 on dist-test-slave-x1lh"
server_key: "b3ae01cdfab9748a20b2cdc376e8d83b"
server_key_iv: "4e127725b5cf91dba8f8c618804aa98d"
server_key_version: "encryptionkey@0"
I20260925 22:58:46.290182 9288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "5c1feb5081ef4a14a40b176c34dff6df"
format_stamp: "Formatted at 2026-09-25 22:58:46 on dist-test-slave-x1lh"
server_key: "b3ae01cdfab9748a20b2cdc376e8d83b"
server_key_iv: "4e127725b5cf91dba8f8c618804aa98d"
server_key_version: "encryptionkey@0"
I20260925 22:58:46.294842 9288 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:58:46.297923 9303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:46.299566 9288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:46.299706 9288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "5c1feb5081ef4a14a40b176c34dff6df"
format_stamp: "Formatted at 2026-09-25 22:58:46 on dist-test-slave-x1lh"
server_key: "b3ae01cdfab9748a20b2cdc376e8d83b"
server_key_iv: "4e127725b5cf91dba8f8c618804aa98d"
server_key_version: "encryptionkey@0"
I20260925 22:58:46.299881 9288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:46.305528 9288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:46.309695 9288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:46.310181 9288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:46.318277 9288 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45209
I20260925 22:58:46.318301 9355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45209 every 8 connection(s)
I20260925 22:58:46.319321 9288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:58:46.322583 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 9288
I20260925 22:58:46.322739 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:58:46.322876 9356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:46.323082 20939 external_mini_cluster.cc:1428] Setting key 99842be7d0935ea00a98e7e95cc2f211
I20260925 22:58:46.328604 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:46.324623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50131 (local address 127.0.0.1:45209)
0925 22:58:46.325128 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:46.325143 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:46.325181 (+ 38us) server_negotiation.cc:407] Connection header received
0925 22:58:46.325846 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:46.325865 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:46.326205 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:46.326562 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:46.327405 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:46.327411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:46.327423 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:46.327434 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:46.327487 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:46.327506 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:46.327656 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:46.327725 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 22:58:46.327884 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":189,"threads_started":1}
I20260925 22:58:46.330207 9356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c1feb5081ef4a14a40b176c34dff6df: Bootstrap starting.
I20260925 22:58:46.331634 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:58:46.333144 9356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c1feb5081ef4a14a40b176c34dff6df: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:46.334123 9356 log.cc:824] T 00000000000000000000000000000000 P 5c1feb5081ef4a14a40b176c34dff6df: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:46.336053 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:46.334174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58191 (local address 127.0.0.1:45209)
0925 22:58:46.334323 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:46.334327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:46.334508 (+ 181us) server_negotiation.cc:407] Connection header received
0925 22:58:46.334652 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:46.334655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:46.334687 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:46.334752 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:46.335623 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:46.335627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:46.335629 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:46.335642 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:46.335671 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:46.335685 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:46.335778 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:46.335880 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 22:58:46.335919 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:58:46.336915 9356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c1feb5081ef4a14a40b176c34dff6df: No bootstrap required, opened a new log
I20260925 22:58:46.340698 9356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c1feb5081ef4a14a40b176c34dff6df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1feb5081ef4a14a40b176c34dff6df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45209 } }
I20260925 22:58:46.341073 9356 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c1feb5081ef4a14a40b176c34dff6df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:46.341156 9356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c1feb5081ef4a14a40b176c34dff6df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1feb5081ef4a14a40b176c34dff6df, State: Initialized, Role: FOLLOWER
I20260925 22:58:46.341632 9356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c1feb5081ef4a14a40b176c34dff6df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1feb5081ef4a14a40b176c34dff6df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45209 } }
I20260925 22:58:46.341758 9356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c1feb5081ef4a14a40b176c34dff6df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:58:46.341804 9356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c1feb5081ef4a14a40b176c34dff6df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:58:46.341935 9356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c1feb5081ef4a14a40b176c34dff6df [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:46.343066 9356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c1feb5081ef4a14a40b176c34dff6df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1feb5081ef4a14a40b176c34dff6df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45209 } }
I20260925 22:58:46.343475 9356 leader_election.cc:302] T 00000000000000000000000000000000 P 5c1feb5081ef4a14a40b176c34dff6df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c1feb5081ef4a14a40b176c34dff6df; no voters:
I20260925 22:58:46.343876 9356 leader_election.cc:288] T 00000000000000000000000000000000 P 5c1feb5081ef4a14a40b176c34dff6df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:58:46.344038 9366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c1feb5081ef4a14a40b176c34dff6df [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:46.344300 9366 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c1feb5081ef4a14a40b176c34dff6df [term 1 LEADER]: Becoming Leader. State: Replica: 5c1feb5081ef4a14a40b176c34dff6df, State: Running, Role: LEADER
I20260925 22:58:46.344630 9366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c1feb5081ef4a14a40b176c34dff6df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1feb5081ef4a14a40b176c34dff6df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45209 } }
I20260925 22:58:46.345296 9356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c1feb5081ef4a14a40b176c34dff6df [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:58:46.346426 9367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c1feb5081ef4a14a40b176c34dff6df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c1feb5081ef4a14a40b176c34dff6df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1feb5081ef4a14a40b176c34dff6df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45209 } } }
I20260925 22:58:46.346565 9367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c1feb5081ef4a14a40b176c34dff6df [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:46.346758 9368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c1feb5081ef4a14a40b176c34dff6df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c1feb5081ef4a14a40b176c34dff6df. Latest consensus state: current_term: 1 leader_uuid: "5c1feb5081ef4a14a40b176c34dff6df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1feb5081ef4a14a40b176c34dff6df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45209 } } }
I20260925 22:58:46.346911 9368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c1feb5081ef4a14a40b176c34dff6df [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:46.347020 9375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:58:46.350494 9375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:58:46.357527 9375 catalog_manager.cc:1347] Generated new cluster ID: 911eb35e5ef948ac8423006e84c14a42
I20260925 22:58:46.357623 9375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:58:46.387315 9375 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:58:46.387704 9375 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:58:46.393909 9375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c1feb5081ef4a14a40b176c34dff6df: Generated new TSK 0
I20260925 22:58:46.394850 9375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:58:46.401084 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 9288
2026-09-25T22:58:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1524 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:58:46 dist-test-slave-x1lh krb5kdc[9387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:58:46 dist-test-slave-x1lh krb5kdc[9387](info): set up 2 sockets
Sep 25 22:58:46 dist-test-slave-x1lh krb5kdc[9387](info): commencing operation
krb5kdc: starting...
W20260925 22:58:47.485275 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:58:47 dist-test-slave-x1lh krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:58:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:58:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:58:47.620651 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42733
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:47.739372 9400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:47.739735 9400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:47.739840 9400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:47.744091 9400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:58:47.744210 9400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:47.744246 9400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:47.744285 9400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:58:47.744320 9400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:58:47.749938 9400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:47.751159 9400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:47.752249 9400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:47.761466 9406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:47.761489 9405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:47.761498 9408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:47.761567 9400 server_base.cc:1034] running on GCE node
I20260925 22:58:47.762478 9400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:47.763733 9400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:47.764930 9400 hybrid_clock.cc:648] HybridClock initialized: now 1790377127764918 us; error 31 us; skew 500 ppm
Sep 25 22:58:47 dist-test-slave-x1lh krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:47.767979 9400 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:58:47.768990 9400 webserver.cc:460] Webserver started at http://127.0.0.1:33037/ using document root <none> and password file <none>
I20260925 22:58:47.769557 9400 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:47.769620 9400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:47.769811 9400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:47.772059 9400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "4832d48b1f4f41c9a1edcfe120448d19"
format_stamp: "Formatted at 2026-09-25 22:58:47 on dist-test-slave-x1lh"
server_key: "b515d6de67e36b19f63bd3d7f9114585"
server_key_iv: "8e945d2e63649dafec3a0cb3ec7900a2"
server_key_version: "encryptionkey@0"
I20260925 22:58:47.772650 9400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "4832d48b1f4f41c9a1edcfe120448d19"
format_stamp: "Formatted at 2026-09-25 22:58:47 on dist-test-slave-x1lh"
server_key: "b515d6de67e36b19f63bd3d7f9114585"
server_key_iv: "8e945d2e63649dafec3a0cb3ec7900a2"
server_key_version: "encryptionkey@0"
I20260925 22:58:47.776818 9400 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:58:47.779577 9415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:47.781360 9400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:47.781543 9400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "4832d48b1f4f41c9a1edcfe120448d19"
format_stamp: "Formatted at 2026-09-25 22:58:47 on dist-test-slave-x1lh"
server_key: "b515d6de67e36b19f63bd3d7f9114585"
server_key_iv: "8e945d2e63649dafec3a0cb3ec7900a2"
server_key_version: "encryptionkey@0"
I20260925 22:58:47.781684 9400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:47.799474 9400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:47.803372 9400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:47.803606 9400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:47.811738 9400 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42733
I20260925 22:58:47.811770 9467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42733 every 8 connection(s)
I20260925 22:58:47.812845 9400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:58:47.816262 9468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:47.818261 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 9400
I20260925 22:58:47.818421 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:58:47.818692 20939 external_mini_cluster.cc:1428] Setting key 9f3ffcf44dc94133dc11f9fdd33b6faf
I20260925 22:58:47.825217 9468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4832d48b1f4f41c9a1edcfe120448d19: Bootstrap starting.
I20260925 22:58:47.828962 9468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4832d48b1f4f41c9a1edcfe120448d19: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:47.830163 9468 log.cc:824] T 00000000000000000000000000000000 P 4832d48b1f4f41c9a1edcfe120448d19: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:58:47 dist-test-slave-x1lh krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:58:47.833108 9468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4832d48b1f4f41c9a1edcfe120448d19: No bootstrap required, opened a new log
I20260925 22:58:47.837473 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:47.820561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:42757 (local address 127.0.0.1:42733)
0925 22:58:47.821140 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:47.821164 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:47.821205 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:58:47.822321 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:47.822351 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:47.822866 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:47.823289 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:47.824805 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:47.826526 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:47.827694 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:47.828289 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:47.828472 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:47.831450 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:47.831455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:47.831471 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:47.831500 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:47.833701 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:47.834355 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:47.834364 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:47.834373 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:47.834553 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:47.835038 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:47.835041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:47.835043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:47.835492 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:47.835717 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:47.836088 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 22:58:47.836368 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":150,"threads_started":1}
I20260925 22:58:47.839393 9468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4832d48b1f4f41c9a1edcfe120448d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4832d48b1f4f41c9a1edcfe120448d19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42733 } }
I20260925 22:58:47.839718 9468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4832d48b1f4f41c9a1edcfe120448d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:47.839812 9468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4832d48b1f4f41c9a1edcfe120448d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4832d48b1f4f41c9a1edcfe120448d19, State: Initialized, Role: FOLLOWER
I20260925 22:58:47.840363 9468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4832d48b1f4f41c9a1edcfe120448d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4832d48b1f4f41c9a1edcfe120448d19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42733 } }
I20260925 22:58:47.840564 9468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4832d48b1f4f41c9a1edcfe120448d19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:58:47.840667 9468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4832d48b1f4f41c9a1edcfe120448d19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:58:47.840791 9468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4832d48b1f4f41c9a1edcfe120448d19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:47.841188 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:58:47.842301 9468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4832d48b1f4f41c9a1edcfe120448d19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4832d48b1f4f41c9a1edcfe120448d19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42733 } }
I20260925 22:58:47.842823 9468 leader_election.cc:302] T 00000000000000000000000000000000 P 4832d48b1f4f41c9a1edcfe120448d19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4832d48b1f4f41c9a1edcfe120448d19; no voters:
I20260925 22:58:47.843300 9468 leader_election.cc:288] T 00000000000000000000000000000000 P 4832d48b1f4f41c9a1edcfe120448d19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:58:47.843410 9477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4832d48b1f4f41c9a1edcfe120448d19 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:47.843684 9477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4832d48b1f4f41c9a1edcfe120448d19 [term 1 LEADER]: Becoming Leader. State: Replica: 4832d48b1f4f41c9a1edcfe120448d19, State: Running, Role: LEADER
I20260925 22:58:47.844105 9477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4832d48b1f4f41c9a1edcfe120448d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4832d48b1f4f41c9a1edcfe120448d19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42733 } }
I20260925 22:58:47.845017 9468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4832d48b1f4f41c9a1edcfe120448d19 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:58:47.846160 9479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4832d48b1f4f41c9a1edcfe120448d19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4832d48b1f4f41c9a1edcfe120448d19. Latest consensus state: current_term: 1 leader_uuid: "4832d48b1f4f41c9a1edcfe120448d19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4832d48b1f4f41c9a1edcfe120448d19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42733 } } }
I20260925 22:58:47.846335 9479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4832d48b1f4f41c9a1edcfe120448d19 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:47.846158 9478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4832d48b1f4f41c9a1edcfe120448d19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4832d48b1f4f41c9a1edcfe120448d19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4832d48b1f4f41c9a1edcfe120448d19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42733 } } }
I20260925 22:58:47.846503 9478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4832d48b1f4f41c9a1edcfe120448d19 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:47.846803 9487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:58:47.850220 9487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:58:47.856933 9487 catalog_manager.cc:1347] Generated new cluster ID: 51323da00025454aba286751dab12093
I20260925 22:58:47.857018 9487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:58:47.857456 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:47.844289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:48789 (local address 127.0.0.1:42733)
0925 22:58:47.844420 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:47.844426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:47.844482 (+ 56us) server_negotiation.cc:407] Connection header received
0925 22:58:47.844656 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:47.844660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:47.844717 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:47.844826 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:47.847737 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:47.848653 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:47.849956 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:47.850375 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:47.850495 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:47.852389 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:47.852393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:47.852396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:47.852426 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:47.854678 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:47.855545 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:47.855549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:47.855551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:47.855644 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:47.856021 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:47.856024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:47.856026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:47.856232 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:47.856407 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:47.857182 (+ 775us) server_negotiation.cc:299] Negotiation successful
0925 22:58:47.857298 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:58:47.865365 9487 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:58:47.866384 9487 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:58:47.875701 9487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4832d48b1f4f41c9a1edcfe120448d19: Generated new TSK 0
I20260925 22:58:47.876662 9487 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:58:47.882781 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 9400
W20260925 22:58:47.888552 9385 connection.cc:535] client connection to 127.0.0.1:42733 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42733): Connection reset by peer (error 104)
2026-09-25T22:58:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1481 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:58:47 dist-test-slave-x1lh krb5kdc[9499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:58:47 dist-test-slave-x1lh krb5kdc[9499](info): set up 2 sockets
Sep 25 22:58:47 dist-test-slave-x1lh krb5kdc[9499](info): commencing operation
krb5kdc: starting...
W20260925 22:58:48.929356 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:58:49 dist-test-slave-x1lh krb5kdc[9499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:58:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:58:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:58:49.059764 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36195
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:49.175469 9512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:49.175827 9512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:49.175906 9512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:49.180284 9512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:58:49.180397 9512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:49.180435 9512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:49.180473 9512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:58:49.180505 9512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:58:49.186458 9512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:49.187697 9512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:49.188792 9512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:49.198586 9520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:49.198503 9517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:49.198503 9518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:49.198532 9512 server_base.cc:1034] running on GCE node
I20260925 22:58:49.199474 9512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:49.200716 9512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:49.201913 9512 hybrid_clock.cc:648] HybridClock initialized: now 1790377129201889 us; error 39 us; skew 500 ppm
Sep 25 22:58:49 dist-test-slave-x1lh krb5kdc[9499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:49.205407 9512 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:58:49.206611 9512 webserver.cc:460] Webserver started at http://127.0.0.1:33547/ using document root <none> and password file <none>
I20260925 22:58:49.207219 9512 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:49.207314 9512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:49.207554 9512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:49.209857 9512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "1da3ebcd280747b79ec5552b2ae5ca88"
format_stamp: "Formatted at 2026-09-25 22:58:49 on dist-test-slave-x1lh"
server_key: "bcc7b5f018a2f50a85da9c2686d405d0"
server_key_iv: "90936a0ce16b1b2c60a811a7795a37b8"
server_key_version: "encryptionkey@0"
I20260925 22:58:49.210471 9512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "1da3ebcd280747b79ec5552b2ae5ca88"
format_stamp: "Formatted at 2026-09-25 22:58:49 on dist-test-slave-x1lh"
server_key: "bcc7b5f018a2f50a85da9c2686d405d0"
server_key_iv: "90936a0ce16b1b2c60a811a7795a37b8"
server_key_version: "encryptionkey@0"
I20260925 22:58:49.214573 9512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:58:49.217366 9527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:49.218734 9512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:49.218906 9512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "1da3ebcd280747b79ec5552b2ae5ca88"
format_stamp: "Formatted at 2026-09-25 22:58:49 on dist-test-slave-x1lh"
server_key: "bcc7b5f018a2f50a85da9c2686d405d0"
server_key_iv: "90936a0ce16b1b2c60a811a7795a37b8"
server_key_version: "encryptionkey@0"
I20260925 22:58:49.219038 9512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:49.235505 9512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:49.239217 9512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:49.239442 9512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:49.247990 9512 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36195
I20260925 22:58:49.248019 9579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36195 every 8 connection(s)
I20260925 22:58:49.249194 9512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:58:49.252404 9580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:49.257280 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 9512
I20260925 22:58:49.257459 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:58:49.257828 20939 external_mini_cluster.cc:1428] Setting key 96ed9fda3288df20aff0b60cacfe2ffa
I20260925 22:58:49.260174 9580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1da3ebcd280747b79ec5552b2ae5ca88: Bootstrap starting.
I20260925 22:58:49.262874 9580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1da3ebcd280747b79ec5552b2ae5ca88: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:49.263793 9580 log.cc:824] T 00000000000000000000000000000000 P 1da3ebcd280747b79ec5552b2ae5ca88: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:49.266340 9580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1da3ebcd280747b79ec5552b2ae5ca88: No bootstrap required, opened a new log
I20260925 22:58:49.268222 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:49.259636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:54115 (local address 127.0.0.1:36195)
0925 22:58:49.260172 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:49.260188 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:49.260237 (+ 49us) server_negotiation.cc:407] Connection header received
0925 22:58:49.261133 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:49.261156 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:49.261559 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:49.261960 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:49.262412 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:49.263523 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:49.264508 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:49.265213 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:49.265408 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:49.266348 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:49.266354 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:49.266369 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:49.266847 (+ 478us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:49.266909 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:49.266933 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:49.267078 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:49.267267 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 22:58:49.267423 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":169,"threads_started":1}
I20260925 22:58:49.271126 9580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1da3ebcd280747b79ec5552b2ae5ca88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da3ebcd280747b79ec5552b2ae5ca88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } }
I20260925 22:58:49.271498 9580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1da3ebcd280747b79ec5552b2ae5ca88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:49.271574 9580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1da3ebcd280747b79ec5552b2ae5ca88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da3ebcd280747b79ec5552b2ae5ca88, State: Initialized, Role: FOLLOWER
I20260925 22:58:49.271943 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:58:49.272140 9580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1da3ebcd280747b79ec5552b2ae5ca88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da3ebcd280747b79ec5552b2ae5ca88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } }
I20260925 22:58:49.272306 9580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1da3ebcd280747b79ec5552b2ae5ca88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:58:49.272383 9580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1da3ebcd280747b79ec5552b2ae5ca88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:58:49.272522 9580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1da3ebcd280747b79ec5552b2ae5ca88 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:49.273764 9580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1da3ebcd280747b79ec5552b2ae5ca88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da3ebcd280747b79ec5552b2ae5ca88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } }
I20260925 22:58:49.274207 9580 leader_election.cc:302] T 00000000000000000000000000000000 P 1da3ebcd280747b79ec5552b2ae5ca88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1da3ebcd280747b79ec5552b2ae5ca88; no voters:
I20260925 22:58:49.274708 9580 leader_election.cc:288] T 00000000000000000000000000000000 P 1da3ebcd280747b79ec5552b2ae5ca88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:58:49.274820 9590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1da3ebcd280747b79ec5552b2ae5ca88 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:49.275065 9590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1da3ebcd280747b79ec5552b2ae5ca88 [term 1 LEADER]: Becoming Leader. State: Replica: 1da3ebcd280747b79ec5552b2ae5ca88, State: Running, Role: LEADER
I20260925 22:58:49.275461 9590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1da3ebcd280747b79ec5552b2ae5ca88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da3ebcd280747b79ec5552b2ae5ca88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } }
I20260925 22:58:49.276301 9580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1da3ebcd280747b79ec5552b2ae5ca88 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:58:49.277293 9592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1da3ebcd280747b79ec5552b2ae5ca88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1da3ebcd280747b79ec5552b2ae5ca88. Latest consensus state: current_term: 1 leader_uuid: "1da3ebcd280747b79ec5552b2ae5ca88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da3ebcd280747b79ec5552b2ae5ca88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } } }
I20260925 22:58:49.277273 9591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1da3ebcd280747b79ec5552b2ae5ca88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1da3ebcd280747b79ec5552b2ae5ca88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da3ebcd280747b79ec5552b2ae5ca88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } } }
I20260925 22:58:49.277495 9591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1da3ebcd280747b79ec5552b2ae5ca88 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:49.277495 9592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1da3ebcd280747b79ec5552b2ae5ca88 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:49.277935 9599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:58:49.280903 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:49.274283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:51123 (local address 127.0.0.1:36195)
0925 22:58:49.274464 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:49.274469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:49.274585 (+ 116us) server_negotiation.cc:407] Connection header received
0925 22:58:49.274688 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:49.274691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:49.274755 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:49.274853 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:49.275357 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:49.276176 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:49.278674 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:49.279006 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:49.279084 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:49.280287 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:49.280291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:49.280294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:49.280312 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:49.280347 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:49.280364 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:49.280475 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:49.280676 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 22:58:49.280719 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:58:49.281656 9599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:58:49.289054 9599 catalog_manager.cc:1347] Generated new cluster ID: 147b92bceef247a1a4c7130786a8c481
I20260925 22:58:49.289149 9599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:58:49.338289 9599 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:58:49.339730 9599 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:58:49.346935 9599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1da3ebcd280747b79ec5552b2ae5ca88: Generated new TSK 0
I20260925 22:58:49.347915 9599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:58:49.363842 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 9512
W20260925 22:58:49.371420 9497 connection.cc:535] client connection to 127.0.0.1:36195 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36195): Connection reset by peer (error 104)
2026-09-25T22:58:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1483 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:58:49 dist-test-slave-x1lh krb5kdc[9611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:58:49 dist-test-slave-x1lh krb5kdc[9611](info): set up 2 sockets
Sep 25 22:58:49 dist-test-slave-x1lh krb5kdc[9611](info): commencing operation
krb5kdc: starting...
W20260925 22:58:50.429240 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:58:50 dist-test-slave-x1lh krb5kdc[9611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:58:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:58:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:58:50.564628 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39985
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:50.691365 9624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:50.691756 9624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:50.691856 9624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:50.696602 9624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:58:50.696753 9624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:50.696807 9624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:50.696856 9624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:58:50.696914 9624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:58:50.703348 9624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:50.704627 9624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:50.705837 9624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:50.715572 9630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:50.715591 9629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:50.715593 9632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:50.716056 9624 server_base.cc:1034] running on GCE node
I20260925 22:58:50.716656 9624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:50.717941 9624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:50.719117 9624 hybrid_clock.cc:648] HybridClock initialized: now 1790377130719087 us; error 45 us; skew 500 ppm
Sep 25 22:58:50 dist-test-slave-x1lh krb5kdc[9611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:50.722194 9624 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:58:50.723207 9624 webserver.cc:460] Webserver started at http://127.0.0.1:34659/ using document root <none> and password file <none>
I20260925 22:58:50.723800 9624 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:50.723893 9624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:50.724120 9624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:50.726300 9624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "88e1ccb063324fcb867b5a5db46a3256"
format_stamp: "Formatted at 2026-09-25 22:58:50 on dist-test-slave-x1lh"
server_key: "3df8cdc699fd98b2517471f035df1bad"
server_key_iv: "f01e37795f980eb46d5f19c2020f348a"
server_key_version: "encryptionkey@0"
I20260925 22:58:50.726898 9624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "88e1ccb063324fcb867b5a5db46a3256"
format_stamp: "Formatted at 2026-09-25 22:58:50 on dist-test-slave-x1lh"
server_key: "3df8cdc699fd98b2517471f035df1bad"
server_key_iv: "f01e37795f980eb46d5f19c2020f348a"
server_key_version: "encryptionkey@0"
I20260925 22:58:50.731014 9624 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:58:50.733805 9639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:50.735208 9624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:50.735414 9624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "88e1ccb063324fcb867b5a5db46a3256"
format_stamp: "Formatted at 2026-09-25 22:58:50 on dist-test-slave-x1lh"
server_key: "3df8cdc699fd98b2517471f035df1bad"
server_key_iv: "f01e37795f980eb46d5f19c2020f348a"
server_key_version: "encryptionkey@0"
I20260925 22:58:50.735545 9624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:50.740382 9624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:50.744019 9624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:50.744247 9624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:50.752096 9624 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39985
I20260925 22:58:50.752486 9691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39985 every 8 connection(s)
I20260925 22:58:50.753163 9624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:58:50.756115 9692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:50.762121 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 9624
I20260925 22:58:50.762253 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:58:50.762912 20939 external_mini_cluster.cc:1428] Setting key 17d2e7ecb3d7b2987b5e5bda1ff53187
I20260925 22:58:50.762897 9692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e1ccb063324fcb867b5a5db46a3256: Bootstrap starting.
I20260925 22:58:50.765758 9692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88e1ccb063324fcb867b5a5db46a3256: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:50.767059 9692 log.cc:824] T 00000000000000000000000000000000 P 88e1ccb063324fcb867b5a5db46a3256: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:50.769239 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:50.764423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:56949 (local address 127.0.0.1:39985)
0925 22:58:50.764970 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:50.764987 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:50.765027 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:58:50.765773 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:50.765792 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:50.766191 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:50.766567 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:50.767598 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:50.767603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:50.767627 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:50.767639 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:50.767703 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:50.767728 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:50.767848 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:50.768035 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 22:58:50.768214 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":152,"threads_started":1}
I20260925 22:58:50.770506 9692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e1ccb063324fcb867b5a5db46a3256: No bootstrap required, opened a new log
I20260925 22:58:50.772219 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:58:50.774857 9692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88e1ccb063324fcb867b5a5db46a3256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e1ccb063324fcb867b5a5db46a3256" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39985 } }
I20260925 22:58:50.775198 9692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88e1ccb063324fcb867b5a5db46a3256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:50.775265 9692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88e1ccb063324fcb867b5a5db46a3256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e1ccb063324fcb867b5a5db46a3256, State: Initialized, Role: FOLLOWER
I20260925 22:58:50.775755 9692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88e1ccb063324fcb867b5a5db46a3256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e1ccb063324fcb867b5a5db46a3256" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39985 } }
I20260925 22:58:50.775946 9692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88e1ccb063324fcb867b5a5db46a3256 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:58:50.776023 9692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88e1ccb063324fcb867b5a5db46a3256 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:58:50.776158 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:50.774475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:39279 (local address 127.0.0.1:39985)
0925 22:58:50.774610 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:50.774614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:50.774740 (+ 126us) server_negotiation.cc:407] Connection header received
0925 22:58:50.774859 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:50.774862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:50.774908 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:50.774995 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:50.775713 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:50.775717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:50.775720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:50.775731 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:50.775759 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:50.775776 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:50.775861 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:50.775983 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 22:58:50.776026 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:58:50.776140 9692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88e1ccb063324fcb867b5a5db46a3256 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:50.777375 9692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88e1ccb063324fcb867b5a5db46a3256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e1ccb063324fcb867b5a5db46a3256" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39985 } }
I20260925 22:58:50.777730 9692 leader_election.cc:302] T 00000000000000000000000000000000 P 88e1ccb063324fcb867b5a5db46a3256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e1ccb063324fcb867b5a5db46a3256; no voters:
I20260925 22:58:50.778047 9692 leader_election.cc:288] T 00000000000000000000000000000000 P 88e1ccb063324fcb867b5a5db46a3256 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:58:50.778252 9702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88e1ccb063324fcb867b5a5db46a3256 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:50.778517 9702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88e1ccb063324fcb867b5a5db46a3256 [term 1 LEADER]: Becoming Leader. State: Replica: 88e1ccb063324fcb867b5a5db46a3256, State: Running, Role: LEADER
I20260925 22:58:50.778919 9702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88e1ccb063324fcb867b5a5db46a3256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e1ccb063324fcb867b5a5db46a3256" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39985 } }
I20260925 22:58:50.779599 9692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88e1ccb063324fcb867b5a5db46a3256 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:58:50.780974 9704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e1ccb063324fcb867b5a5db46a3256 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e1ccb063324fcb867b5a5db46a3256. Latest consensus state: current_term: 1 leader_uuid: "88e1ccb063324fcb867b5a5db46a3256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e1ccb063324fcb867b5a5db46a3256" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39985 } } }
I20260925 22:58:50.781009 9703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e1ccb063324fcb867b5a5db46a3256 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88e1ccb063324fcb867b5a5db46a3256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e1ccb063324fcb867b5a5db46a3256" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39985 } } }
I20260925 22:58:50.781222 9703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e1ccb063324fcb867b5a5db46a3256 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:50.781255 9704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e1ccb063324fcb867b5a5db46a3256 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:50.781702 9711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:58:50.785215 9711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:58:50.792281 9711 catalog_manager.cc:1347] Generated new cluster ID: 6ec8d7c9a2754c64bb21a53cad4ff8fa
I20260925 22:58:50.792407 9711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:58:50.813338 9711 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:58:50.813719 9711 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:58:50.829085 9711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88e1ccb063324fcb867b5a5db46a3256: Generated new TSK 0
I20260925 22:58:50.829918 9711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:58:50.832935 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 9624
2026-09-25T22:58:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1466 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:58:50 dist-test-slave-x1lh krb5kdc[9723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:58:50 dist-test-slave-x1lh krb5kdc[9723](info): set up 2 sockets
Sep 25 22:58:50 dist-test-slave-x1lh krb5kdc[9723](info): commencing operation
krb5kdc: starting...
W20260925 22:58:51.913378 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:58:51 dist-test-slave-x1lh krb5kdc[9723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:58:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:58:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:58:52.048174 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37097
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:52.164351 9736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:52.164645 9736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:52.164687 9736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:52.168977 9736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:58:52.169054 9736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:52.169073 9736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:52.169090 9736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:58:52.169133 9736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:58:52.174780 9736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37097
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:52.175954 9736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:52.177202 9736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:52.189814 9742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:52.189805 9741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:52.190088 9736 server_base.cc:1034] running on GCE node
W20260925 22:58:52.189805 9744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:52.190886 9736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:52.192224 9736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:52.193437 9736 hybrid_clock.cc:648] HybridClock initialized: now 1790377132193420 us; error 34 us; skew 500 ppm
Sep 25 22:58:52 dist-test-slave-x1lh krb5kdc[9723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:52.201157 9736 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:58:52.202296 9736 webserver.cc:460] Webserver started at http://127.0.0.1:44729/ using document root <none> and password file <none>
I20260925 22:58:52.202912 9736 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:52.202981 9736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:52.203198 9736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:52.205494 9736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "fd5d520134b74bf69d54e64aa8b059d2"
format_stamp: "Formatted at 2026-09-25 22:58:52 on dist-test-slave-x1lh"
server_key: "1b4effecbcbef7e1bb644d0733ed9010"
server_key_iv: "214c24333185342eacf8eb68d471f742"
server_key_version: "encryptionkey@0"
I20260925 22:58:52.206167 9736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "fd5d520134b74bf69d54e64aa8b059d2"
format_stamp: "Formatted at 2026-09-25 22:58:52 on dist-test-slave-x1lh"
server_key: "1b4effecbcbef7e1bb644d0733ed9010"
server_key_iv: "214c24333185342eacf8eb68d471f742"
server_key_version: "encryptionkey@0"
I20260925 22:58:52.210242 9736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:58:52.212841 9751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:52.214285 9736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:52.214454 9736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "fd5d520134b74bf69d54e64aa8b059d2"
format_stamp: "Formatted at 2026-09-25 22:58:52 on dist-test-slave-x1lh"
server_key: "1b4effecbcbef7e1bb644d0733ed9010"
server_key_iv: "214c24333185342eacf8eb68d471f742"
server_key_version: "encryptionkey@0"
I20260925 22:58:52.214582 9736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:52.226667 9736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:52.230026 9736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:52.230244 9736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:52.237836 9736 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37097
I20260925 22:58:52.237869 9803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37097 every 8 connection(s)
I20260925 22:58:52.238858 9736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:58:52.241914 9804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:52.245646 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 9736
I20260925 22:58:52.245766 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:58:52.246125 20939 external_mini_cluster.cc:1428] Setting key 3164d5c69694ddcb914e672d19c7ba3a
I20260925 22:58:52.248195 9804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd5d520134b74bf69d54e64aa8b059d2: Bootstrap starting.
I20260925 22:58:52.251142 9804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd5d520134b74bf69d54e64aa8b059d2: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:52.252089 9804 log.cc:824] T 00000000000000000000000000000000 P fd5d520134b74bf69d54e64aa8b059d2: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:52.254622 9804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd5d520134b74bf69d54e64aa8b059d2: No bootstrap required, opened a new log
Sep 25 22:58:52 dist-test-slave-x1lh krb5kdc[9723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:58:52.259119 9804 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd5d520134b74bf69d54e64aa8b059d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd5d520134b74bf69d54e64aa8b059d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } }
I20260925 22:58:52.259439 9804 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd5d520134b74bf69d54e64aa8b059d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:52.259510 9804 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd5d520134b74bf69d54e64aa8b059d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5d520134b74bf69d54e64aa8b059d2, State: Initialized, Role: FOLLOWER
I20260925 22:58:52.260052 9804 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd5d520134b74bf69d54e64aa8b059d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd5d520134b74bf69d54e64aa8b059d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } }
I20260925 22:58:52.260211 9804 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd5d520134b74bf69d54e64aa8b059d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:58:52.260298 9804 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd5d520134b74bf69d54e64aa8b059d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:58:52.260402 9804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd5d520134b74bf69d54e64aa8b059d2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:52.261519 9804 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd5d520134b74bf69d54e64aa8b059d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd5d520134b74bf69d54e64aa8b059d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } }
I20260925 22:58:52.261994 9804 leader_election.cc:302] T 00000000000000000000000000000000 P fd5d520134b74bf69d54e64aa8b059d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd5d520134b74bf69d54e64aa8b059d2; no voters:
I20260925 22:58:52.262418 9804 leader_election.cc:288] T 00000000000000000000000000000000 P fd5d520134b74bf69d54e64aa8b059d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:58:52.262494 9809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd5d520134b74bf69d54e64aa8b059d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:52.262684 9809 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd5d520134b74bf69d54e64aa8b059d2 [term 1 LEADER]: Becoming Leader. State: Replica: fd5d520134b74bf69d54e64aa8b059d2, State: Running, Role: LEADER
I20260925 22:58:52.262724 9807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:52.247672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:38987 (local address 127.0.0.1:37097)
0925 22:58:52.248140 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:52.248160 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:52.248204 (+ 44us) server_negotiation.cc:407] Connection header received
0925 22:58:52.249127 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:52.249153 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:52.249647 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:52.250053 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:52.250890 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:52.252446 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:52.253580 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:52.254311 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:52.254476 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:52.257055 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:52.257062 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:52.257087 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:52.257127 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:52.259307 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:52.259960 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:52.259966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:52.259971 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:52.260090 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:52.260444 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:52.260452 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:52.260453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:52.260981 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:52.261224 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:52.261538 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 22:58:52.261778 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":151,"threads_started":1}
I20260925 22:58:52.263209 9809 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd5d520134b74bf69d54e64aa8b059d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd5d520134b74bf69d54e64aa8b059d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } }
I20260925 22:58:52.263809 9804 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd5d520134b74bf69d54e64aa8b059d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:58:52.264958 9810 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd5d520134b74bf69d54e64aa8b059d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd5d520134b74bf69d54e64aa8b059d2. Latest consensus state: current_term: 1 leader_uuid: "fd5d520134b74bf69d54e64aa8b059d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd5d520134b74bf69d54e64aa8b059d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } } }
I20260925 22:58:52.265367 9810 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd5d520134b74bf69d54e64aa8b059d2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:52.265264 9811 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd5d520134b74bf69d54e64aa8b059d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd5d520134b74bf69d54e64aa8b059d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd5d520134b74bf69d54e64aa8b059d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } } }
I20260925 22:58:52.265684 9811 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd5d520134b74bf69d54e64aa8b059d2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:52.266121 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:58:52.266650 9819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:58:52.269896 9819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:58:52.276583 9807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:52.268147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:35901 (local address 127.0.0.1:37097)
0925 22:58:52.268262 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:52.268266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:52.268490 (+ 224us) server_negotiation.cc:407] Connection header received
0925 22:58:52.268638 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:52.268641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:52.268693 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:52.268774 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:52.269488 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:52.270066 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:52.270904 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:52.271158 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:52.271263 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:52.272707 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:52.272711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:52.272714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:52.272742 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:52.274288 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:52.274993 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:52.274997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:52.274999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:52.275088 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:52.275500 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:52.275503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:52.275505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:52.275731 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:52.275858 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:52.276309 (+ 451us) server_negotiation.cc:299] Negotiation successful
0925 22:58:52.276421 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:58:52.276895 9819 catalog_manager.cc:1347] Generated new cluster ID: a2c0ed2fd3fe47c4bd375f364bf9a2d4
I20260925 22:58:52.277050 9819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:58:52.286115 9819 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:58:52.287106 9819 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:58:52.293674 9819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd5d520134b74bf69d54e64aa8b059d2: Generated new TSK 0
I20260925 22:58:52.294514 9819 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:58:52.300933 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 9736
W20260925 22:58:52.306260 9721 connection.cc:535] client connection to 127.0.0.1:37097 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37097): Connection reset by peer (error 104)
2026-09-25T22:58:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1467 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:58:52 dist-test-slave-x1lh krb5kdc[9835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:58:52 dist-test-slave-x1lh krb5kdc[9835](info): set up 2 sockets
Sep 25 22:58:52 dist-test-slave-x1lh krb5kdc[9835](info): commencing operation
krb5kdc: starting...
W20260925 22:58:53.361469 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:58:53 dist-test-slave-x1lh krb5kdc[9835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:58:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:58:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:58:53.497447 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37273
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:53.616575 9848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:53.616997 9848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:53.617112 9848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:53.621451 9848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:58:53.621573 9848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:53.621614 9848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:53.621652 9848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:58:53.621690 9848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:58:53.627940 9848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37273
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:53.629209 9848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:53.630352 9848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:58:53.639701 9848 server_base.cc:1034] running on GCE node
W20260925 22:58:53.639643 9856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:53.639643 9854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:53.639643 9853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:53.640532 9848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:53.641757 9848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:53.642936 9848 hybrid_clock.cc:648] HybridClock initialized: now 1790377133642923 us; error 27 us; skew 500 ppm
Sep 25 22:58:53 dist-test-slave-x1lh krb5kdc[9835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:53.646041 9848 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:58:53.647014 9848 webserver.cc:460] Webserver started at http://127.0.0.1:38225/ using document root <none> and password file <none>
I20260925 22:58:53.647609 9848 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:53.647701 9848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:53.647953 9848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:53.650722 9848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "aade4b85f8c2470aadb845187dce79e8"
format_stamp: "Formatted at 2026-09-25 22:58:53 on dist-test-slave-x1lh"
server_key: "e977e18908948247ef4a078a8d3d1523"
server_key_iv: "37036e778d9e11ddb9e7aa0588ed563b"
server_key_version: "encryptionkey@0"
I20260925 22:58:53.651368 9848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "aade4b85f8c2470aadb845187dce79e8"
format_stamp: "Formatted at 2026-09-25 22:58:53 on dist-test-slave-x1lh"
server_key: "e977e18908948247ef4a078a8d3d1523"
server_key_iv: "37036e778d9e11ddb9e7aa0588ed563b"
server_key_version: "encryptionkey@0"
I20260925 22:58:53.655666 9848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 22:58:53.658802 9863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:53.660224 9848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:58:53.660408 9848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "aade4b85f8c2470aadb845187dce79e8"
format_stamp: "Formatted at 2026-09-25 22:58:53 on dist-test-slave-x1lh"
server_key: "e977e18908948247ef4a078a8d3d1523"
server_key_iv: "37036e778d9e11ddb9e7aa0588ed563b"
server_key_version: "encryptionkey@0"
I20260925 22:58:53.660545 9848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:53.676168 9848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:53.679806 9848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:53.680040 9848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:53.687465 9848 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37273
I20260925 22:58:53.687600 9915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37273 every 8 connection(s)
I20260925 22:58:53.688627 9848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:58:53.691884 9916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:53.695259 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 9848
I20260925 22:58:53.695390 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:58:53.695714 20939 external_mini_cluster.cc:1428] Setting key c35dcba322bea86dc5602da0a7173f09
I20260925 22:58:53.698684 9916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aade4b85f8c2470aadb845187dce79e8: Bootstrap starting.
I20260925 22:58:53.701208 9916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aade4b85f8c2470aadb845187dce79e8: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:53.702440 9916 log.cc:824] T 00000000000000000000000000000000 P aade4b85f8c2470aadb845187dce79e8: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:53.704663 9916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aade4b85f8c2470aadb845187dce79e8: No bootstrap required, opened a new log
I20260925 22:58:53.705926 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:53.697347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:60367 (local address 127.0.0.1:37273)
0925 22:58:53.697895 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:53.697912 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:53.697952 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:58:53.698704 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:53.698726 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:53.699158 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:53.699543 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:53.699994 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:53.701112 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:53.702409 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:53.703004 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:53.703203 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:53.704165 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:53.704170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:53.704185 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:53.704592 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:53.704650 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:53.704673 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:53.704819 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:53.705010 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 22:58:53.705168 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":170,"threads_started":1}
I20260925 22:58:53.708832 9916 raft_consensus.cc:348] T 00000000000000000000000000000000 P aade4b85f8c2470aadb845187dce79e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aade4b85f8c2470aadb845187dce79e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37273 } }
I20260925 22:58:53.709198 9916 raft_consensus.cc:374] T 00000000000000000000000000000000 P aade4b85f8c2470aadb845187dce79e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:53.709246 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:58:53.709259 9916 raft_consensus.cc:729] T 00000000000000000000000000000000 P aade4b85f8c2470aadb845187dce79e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aade4b85f8c2470aadb845187dce79e8, State: Initialized, Role: FOLLOWER
I20260925 22:58:53.709810 9916 consensus_queue.cc:260] T 00000000000000000000000000000000 P aade4b85f8c2470aadb845187dce79e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aade4b85f8c2470aadb845187dce79e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37273 } }
I20260925 22:58:53.709990 9916 raft_consensus.cc:388] T 00000000000000000000000000000000 P aade4b85f8c2470aadb845187dce79e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:58:53.710048 9916 raft_consensus.cc:482] T 00000000000000000000000000000000 P aade4b85f8c2470aadb845187dce79e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:58:53.710147 9916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aade4b85f8c2470aadb845187dce79e8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:53.711522 9916 raft_consensus.cc:504] T 00000000000000000000000000000000 P aade4b85f8c2470aadb845187dce79e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aade4b85f8c2470aadb845187dce79e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37273 } }
I20260925 22:58:53.711915 9916 leader_election.cc:302] T 00000000000000000000000000000000 P aade4b85f8c2470aadb845187dce79e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aade4b85f8c2470aadb845187dce79e8; no voters:
I20260925 22:58:53.712313 9916 leader_election.cc:288] T 00000000000000000000000000000000 P aade4b85f8c2470aadb845187dce79e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:58:53.712370 9926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aade4b85f8c2470aadb845187dce79e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:53.712575 9926 raft_consensus.cc:686] T 00000000000000000000000000000000 P aade4b85f8c2470aadb845187dce79e8 [term 1 LEADER]: Becoming Leader. State: Replica: aade4b85f8c2470aadb845187dce79e8, State: Running, Role: LEADER
I20260925 22:58:53.712939 9926 consensus_queue.cc:237] T 00000000000000000000000000000000 P aade4b85f8c2470aadb845187dce79e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aade4b85f8c2470aadb845187dce79e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37273 } }
I20260925 22:58:53.713768 9916 sys_catalog.cc:564] T 00000000000000000000000000000000 P aade4b85f8c2470aadb845187dce79e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:58:53.714202 9926 sys_catalog.cc:455] T 00000000000000000000000000000000 P aade4b85f8c2470aadb845187dce79e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aade4b85f8c2470aadb845187dce79e8. Latest consensus state: current_term: 1 leader_uuid: "aade4b85f8c2470aadb845187dce79e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aade4b85f8c2470aadb845187dce79e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37273 } } }
I20260925 22:58:53.714390 9926 sys_catalog.cc:458] T 00000000000000000000000000000000 P aade4b85f8c2470aadb845187dce79e8 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:53.714648 9926 sys_catalog.cc:455] T 00000000000000000000000000000000 P aade4b85f8c2470aadb845187dce79e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aade4b85f8c2470aadb845187dce79e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aade4b85f8c2470aadb845187dce79e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37273 } } }
I20260925 22:58:53.714811 9926 sys_catalog.cc:458] T 00000000000000000000000000000000 P aade4b85f8c2470aadb845187dce79e8 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:53.715137 9931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:58:53.716846 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:53.711697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:35595 (local address 127.0.0.1:37273)
0925 22:58:53.711826 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:53.711831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:53.712216 (+ 385us) server_negotiation.cc:407] Connection header received
0925 22:58:53.712389 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:53.712395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:53.712456 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:53.712529 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:53.712913 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:53.713528 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:53.714667 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:53.714997 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:53.715078 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:53.716225 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:53.716229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:53.716231 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:53.716246 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:53.716277 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:53.716306 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:53.716369 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:53.716654 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 22:58:53.716692 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:58:53.718767 9931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:58:53.726109 9931 catalog_manager.cc:1347] Generated new cluster ID: 8cecfe3f30034629aaa87e27f4fd06b2
I20260925 22:58:53.726226 9931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:58:53.754444 9931 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:58:53.755913 9931 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:58:53.763074 9931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aade4b85f8c2470aadb845187dce79e8: Generated new TSK 0
I20260925 22:58:53.764081 9931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:58:53.778471 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 9848
W20260925 22:58:53.784341 9833 connection.cc:535] client connection to 127.0.0.1:37273 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37273): Connection reset by peer (error 104)
2026-09-25T22:58:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1478 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:58:53 dist-test-slave-x1lh krb5kdc[9947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:58:53 dist-test-slave-x1lh krb5kdc[9947](info): set up 2 sockets
Sep 25 22:58:53 dist-test-slave-x1lh krb5kdc[9947](info): commencing operation
krb5kdc: starting...
W20260925 22:58:54.865476 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:58:54 dist-test-slave-x1lh krb5kdc[9947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:58:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:58:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:58:54.996536 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:55.113682 9960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:55.114008 9960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:55.114054 9960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:55.118316 9960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:58:55.118400 9960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:55.118420 9960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:55.118438 9960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:58:55.118454 9960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:58:55.124264 9960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38257
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:55.125458 9960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:55.126471 9960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:55.136261 9968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:55.136281 9966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:55.136343 9960 server_base.cc:1034] running on GCE node
W20260925 22:58:55.136289 9965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:55.137202 9960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:55.138424 9960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:55.139626 9960 hybrid_clock.cc:648] HybridClock initialized: now 1790377135139596 us; error 48 us; skew 500 ppm
Sep 25 22:58:55 dist-test-slave-x1lh krb5kdc[9947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:55.142727 9960 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:58:55.143767 9960 webserver.cc:460] Webserver started at http://127.0.0.1:36363/ using document root <none> and password file <none>
I20260925 22:58:55.144383 9960 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:55.144479 9960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:55.144722 9960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:55.147020 9960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "34317ad82c59411f80438f00ad0e9aa5"
format_stamp: "Formatted at 2026-09-25 22:58:55 on dist-test-slave-x1lh"
server_key: "38b6079d813f4b081e5ac61b5e122c6b"
server_key_iv: "c0e627dd8c2f6cba00c1d9ecaa44cfe2"
server_key_version: "encryptionkey@0"
I20260925 22:58:55.147635 9960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "34317ad82c59411f80438f00ad0e9aa5"
format_stamp: "Formatted at 2026-09-25 22:58:55 on dist-test-slave-x1lh"
server_key: "38b6079d813f4b081e5ac61b5e122c6b"
server_key_iv: "c0e627dd8c2f6cba00c1d9ecaa44cfe2"
server_key_version: "encryptionkey@0"
I20260925 22:58:55.151911 9960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:58:55.154892 9975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:55.156438 9960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:58:55.156635 9960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "34317ad82c59411f80438f00ad0e9aa5"
format_stamp: "Formatted at 2026-09-25 22:58:55 on dist-test-slave-x1lh"
server_key: "38b6079d813f4b081e5ac61b5e122c6b"
server_key_iv: "c0e627dd8c2f6cba00c1d9ecaa44cfe2"
server_key_version: "encryptionkey@0"
I20260925 22:58:55.156782 9960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:55.161935 9960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:55.165587 9960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:55.165815 9960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:55.173448 9960 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38257
I20260925 22:58:55.173473 10027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38257 every 8 connection(s)
I20260925 22:58:55.174531 9960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:58:55.178926 10028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:55.183977 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 9960
I20260925 22:58:55.184130 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:58:55.184468 20939 external_mini_cluster.cc:1428] Setting key 129c2db7ab1561223470ec3174380641
I20260925 22:58:55.187848 10028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34317ad82c59411f80438f00ad0e9aa5: Bootstrap starting.
I20260925 22:58:55.190915 10031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:55.186068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:53563 (local address 127.0.0.1:38257)
0925 22:58:55.186562 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:55.186586 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:55.186641 (+ 55us) server_negotiation.cc:407] Connection header received
0925 22:58:55.187575 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:55.187603 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:55.188044 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:55.188482 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:55.189292 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:55.189300 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:55.189332 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:55.189349 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:55.189425 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:55.189459 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:55.189614 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:55.189689 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 22:58:55.189891 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":130,"threads_started":1}
I20260925 22:58:55.191094 10028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34317ad82c59411f80438f00ad0e9aa5: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:55.191932 10028 log.cc:824] T 00000000000000000000000000000000 P 34317ad82c59411f80438f00ad0e9aa5: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:55.194610 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:58:55.194648 10028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34317ad82c59411f80438f00ad0e9aa5: No bootstrap required, opened a new log
I20260925 22:58:55.199677 10031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:55.197418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:60313 (local address 127.0.0.1:38257)
0925 22:58:55.197549 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:55.197554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:55.197849 (+ 295us) server_negotiation.cc:407] Connection header received
0925 22:58:55.198009 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:55.198015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:55.198065 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:55.198176 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:55.199178 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:55.199182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:55.199185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:55.199197 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:55.199226 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:55.199239 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:55.199324 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:55.199447 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 22:58:55.199490 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:58:55.200562 10028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34317ad82c59411f80438f00ad0e9aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34317ad82c59411f80438f00ad0e9aa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38257 } }
I20260925 22:58:55.200946 10028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34317ad82c59411f80438f00ad0e9aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:55.201031 10028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34317ad82c59411f80438f00ad0e9aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34317ad82c59411f80438f00ad0e9aa5, State: Initialized, Role: FOLLOWER
I20260925 22:58:55.201525 10028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34317ad82c59411f80438f00ad0e9aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34317ad82c59411f80438f00ad0e9aa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38257 } }
I20260925 22:58:55.201697 10028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34317ad82c59411f80438f00ad0e9aa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:58:55.201776 10028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34317ad82c59411f80438f00ad0e9aa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:58:55.201896 10028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34317ad82c59411f80438f00ad0e9aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:55.203008 10028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34317ad82c59411f80438f00ad0e9aa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34317ad82c59411f80438f00ad0e9aa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38257 } }
I20260925 22:58:55.203423 10028 leader_election.cc:302] T 00000000000000000000000000000000 P 34317ad82c59411f80438f00ad0e9aa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34317ad82c59411f80438f00ad0e9aa5; no voters:
I20260925 22:58:55.203802 10028 leader_election.cc:288] T 00000000000000000000000000000000 P 34317ad82c59411f80438f00ad0e9aa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:58:55.204057 10038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34317ad82c59411f80438f00ad0e9aa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:55.204403 10038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34317ad82c59411f80438f00ad0e9aa5 [term 1 LEADER]: Becoming Leader. State: Replica: 34317ad82c59411f80438f00ad0e9aa5, State: Running, Role: LEADER
I20260925 22:58:55.204748 10038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34317ad82c59411f80438f00ad0e9aa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34317ad82c59411f80438f00ad0e9aa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38257 } }
I20260925 22:58:55.205389 10028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34317ad82c59411f80438f00ad0e9aa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:58:55.206647 10040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34317ad82c59411f80438f00ad0e9aa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34317ad82c59411f80438f00ad0e9aa5. Latest consensus state: current_term: 1 leader_uuid: "34317ad82c59411f80438f00ad0e9aa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34317ad82c59411f80438f00ad0e9aa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38257 } } }
I20260925 22:58:55.206800 10040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34317ad82c59411f80438f00ad0e9aa5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:55.207154 10039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34317ad82c59411f80438f00ad0e9aa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34317ad82c59411f80438f00ad0e9aa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34317ad82c59411f80438f00ad0e9aa5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38257 } } }
I20260925 22:58:55.207263 10047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:58:55.207333 10039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34317ad82c59411f80438f00ad0e9aa5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:55.211539 10047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:58:55.219254 10047 catalog_manager.cc:1347] Generated new cluster ID: f70fa33a05a1417b87924f8b694e7833
I20260925 22:58:55.219358 10047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:58:55.253512 10047 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:58:55.253909 10047 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:58:55.259044 10047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34317ad82c59411f80438f00ad0e9aa5: Generated new TSK 0
I20260925 22:58:55.259975 10047 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:58:55.272509 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 9960
2026-09-25T22:58:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1494 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:58:55 dist-test-slave-x1lh krb5kdc[10059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:58:55 dist-test-slave-x1lh krb5kdc[10059](info): set up 2 sockets
Sep 25 22:58:55 dist-test-slave-x1lh krb5kdc[10059](info): commencing operation
krb5kdc: starting...
W20260925 22:58:56.377331 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:58:56 dist-test-slave-x1lh krb5kdc[10059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:58:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:58:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:58:56.522370 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:38403
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:44737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:38403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:56.652145 10072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:56.652496 10072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:56.652549 10072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:56.658264 10072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:58:56.658368 10072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:56.658389 10072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:56.658408 10072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:58:56.658432 10072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:58:56.665378 10072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:44737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:38403
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:38403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:56.666651 10072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:56.667794 10072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:56.677450 10078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:56.677446 10077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:56.677446 10080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:56.677873 10072 server_base.cc:1034] running on GCE node
I20260925 22:58:56.678617 10072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:56.680320 10072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:56.681526 10072 hybrid_clock.cc:648] HybridClock initialized: now 1790377136681490 us; error 55 us; skew 500 ppm
Sep 25 22:58:56 dist-test-slave-x1lh krb5kdc[10059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377136, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:56.685070 10072 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:58:56.686256 10072 webserver.cc:460] Webserver started at http://127.20.114.254:42953/ using document root <none> and password file <none>
I20260925 22:58:56.686895 10072 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:56.687003 10072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:56.687252 10072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:56.689757 10072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "fddd33e6373f4c09ae511fb085df5ef2"
format_stamp: "Formatted at 2026-09-25 22:58:56 on dist-test-slave-x1lh"
server_key: "0c7d48bf7a934c576d48050764551828"
server_key_iv: "69d9acb85d835356156ff3f957c8b775"
server_key_version: "encryptionkey@0"
I20260925 22:58:56.690452 10072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "fddd33e6373f4c09ae511fb085df5ef2"
format_stamp: "Formatted at 2026-09-25 22:58:56 on dist-test-slave-x1lh"
server_key: "0c7d48bf7a934c576d48050764551828"
server_key_iv: "69d9acb85d835356156ff3f957c8b775"
server_key_version: "encryptionkey@0"
I20260925 22:58:56.694828 10072 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 22:58:56.697907 10087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:56.699756 10072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:56.699957 10072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "fddd33e6373f4c09ae511fb085df5ef2"
format_stamp: "Formatted at 2026-09-25 22:58:56 on dist-test-slave-x1lh"
server_key: "0c7d48bf7a934c576d48050764551828"
server_key_iv: "69d9acb85d835356156ff3f957c8b775"
server_key_version: "encryptionkey@0"
I20260925 22:58:56.700109 10072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:56.725811 10072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:56.730290 10072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:56.730549 10072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:56.739719 10139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:38403 every 8 connection(s)
I20260925 22:58:56.739755 10072 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:38403
I20260925 22:58:56.741245 10072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:58:56.744477 10140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:56.750699 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 10072
I20260925 22:58:56.751219 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:58:56.751550 20939 external_mini_cluster.cc:1428] Setting key 2657629550b9667d47622f2d4e7f3202
I20260925 22:58:56.751848 10140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fddd33e6373f4c09ae511fb085df5ef2: Bootstrap starting.
I20260925 22:58:56.755174 10140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fddd33e6373f4c09ae511fb085df5ef2: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:56.756242 10140 log.cc:824] T 00000000000000000000000000000000 P fddd33e6373f4c09ae511fb085df5ef2: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:56.758944 10140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fddd33e6373f4c09ae511fb085df5ef2: No bootstrap required, opened a new log
Sep 25 22:58:56 dist-test-slave-x1lh krb5kdc[10059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:58:56.765048 10140 raft_consensus.cc:348] T 00000000000000000000000000000000 P fddd33e6373f4c09ae511fb085df5ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddd33e6373f4c09ae511fb085df5ef2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38403 } }
I20260925 22:58:56.765357 10140 raft_consensus.cc:374] T 00000000000000000000000000000000 P fddd33e6373f4c09ae511fb085df5ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:56.765421 10140 raft_consensus.cc:729] T 00000000000000000000000000000000 P fddd33e6373f4c09ae511fb085df5ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fddd33e6373f4c09ae511fb085df5ef2, State: Initialized, Role: FOLLOWER
I20260925 22:58:56.765933 10140 consensus_queue.cc:260] T 00000000000000000000000000000000 P fddd33e6373f4c09ae511fb085df5ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddd33e6373f4c09ae511fb085df5ef2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38403 } }
I20260925 22:58:56.766072 10140 raft_consensus.cc:388] T 00000000000000000000000000000000 P fddd33e6373f4c09ae511fb085df5ef2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:58:56.766165 10140 raft_consensus.cc:482] T 00000000000000000000000000000000 P fddd33e6373f4c09ae511fb085df5ef2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:58:56.766294 10140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fddd33e6373f4c09ae511fb085df5ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:56.767489 10140 raft_consensus.cc:504] T 00000000000000000000000000000000 P fddd33e6373f4c09ae511fb085df5ef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddd33e6373f4c09ae511fb085df5ef2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38403 } }
I20260925 22:58:56.767920 10140 leader_election.cc:302] T 00000000000000000000000000000000 P fddd33e6373f4c09ae511fb085df5ef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fddd33e6373f4c09ae511fb085df5ef2; no voters:
I20260925 22:58:56.768375 10140 leader_election.cc:288] T 00000000000000000000000000000000 P fddd33e6373f4c09ae511fb085df5ef2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:58:56.768591 10145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fddd33e6373f4c09ae511fb085df5ef2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:56.769023 10145 raft_consensus.cc:686] T 00000000000000000000000000000000 P fddd33e6373f4c09ae511fb085df5ef2 [term 1 LEADER]: Becoming Leader. State: Replica: fddd33e6373f4c09ae511fb085df5ef2, State: Running, Role: LEADER
I20260925 22:58:56.769757 10145 consensus_queue.cc:237] T 00000000000000000000000000000000 P fddd33e6373f4c09ae511fb085df5ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddd33e6373f4c09ae511fb085df5ef2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38403 } }
I20260925 22:58:56.770224 10140 sys_catalog.cc:564] T 00000000000000000000000000000000 P fddd33e6373f4c09ae511fb085df5ef2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:58:56.771965 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:56.753476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55467 (local address 127.20.114.254:38403)
0925 22:58:56.754055 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:56.754073 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:56.754115 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:58:56.754908 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:56.754929 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:56.755416 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:56.755753 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:56.756841 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:56.758103 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:56.759579 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:56.760210 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:56.760356 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:56.764389 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:56.764396 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:56.764425 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:56.764467 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:56.766908 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:56.767726 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:56.767740 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:56.767747 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:56.767904 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:56.768446 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:56.768452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:56.768454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:56.769176 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:56.769690 (+ 514us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:56.770177 (+ 487us) server_negotiation.cc:299] Negotiation successful
0925 22:58:56.770610 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":190,"threads_started":1}
I20260925 22:58:56.772089 10146 sys_catalog.cc:455] T 00000000000000000000000000000000 P fddd33e6373f4c09ae511fb085df5ef2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fddd33e6373f4c09ae511fb085df5ef2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddd33e6373f4c09ae511fb085df5ef2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38403 } } }
I20260925 22:58:56.772461 10146 sys_catalog.cc:458] T 00000000000000000000000000000000 P fddd33e6373f4c09ae511fb085df5ef2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:56.771960 10147 sys_catalog.cc:455] T 00000000000000000000000000000000 P fddd33e6373f4c09ae511fb085df5ef2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fddd33e6373f4c09ae511fb085df5ef2. Latest consensus state: current_term: 1 leader_uuid: "fddd33e6373f4c09ae511fb085df5ef2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddd33e6373f4c09ae511fb085df5ef2" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 38403 } } }
I20260925 22:58:56.772711 10147 sys_catalog.cc:458] T 00000000000000000000000000000000 P fddd33e6373f4c09ae511fb085df5ef2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:56.772891 10158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:58:56.776376 10158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:58:56.776746 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:58:56.783401 10158 catalog_manager.cc:1347] Generated new cluster ID: 68ecb8fbfc234d03ba642c88ee279fed
I20260925 22:58:56.783499 10158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:58:56.791683 10158 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:58:56.792136 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:56.779694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33269 (local address 127.20.114.254:38403)
0925 22:58:56.779814 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:56.779820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:56.780475 (+ 655us) server_negotiation.cc:407] Connection header received
0925 22:58:56.780901 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:56.780905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:56.780970 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:56.781066 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:56.782093 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:56.783505 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:56.785043 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:56.785478 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:56.785595 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:56.787480 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:56.787484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:56.787487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:58:56.787519 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:56.789583 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:56.790269 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:56.790273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:56.790275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:56.790374 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:58:56.791112 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:58:56.791114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:58:56.791116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:58:56.791342 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:56.791488 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:56.791850 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 22:58:56.791974 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:58:56.792986 10158 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:58:56.798627 10158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fddd33e6373f4c09ae511fb085df5ef2: Generated new TSK 0
I20260925 22:58:56.799568 10158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:58:56.807611 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 10072
W20260925 22:58:56.813392 10057 connection.cc:535] client connection to 127.20.114.254:38403 recv error: Network error: failed to read from TLS socket (remote: 127.20.114.254:38403): Connection reset by peer (error 104)
2026-09-25T22:58:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1535 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:58:56 dist-test-slave-x1lh krb5kdc[10171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:58:56 dist-test-slave-x1lh krb5kdc[10171](info): set up 2 sockets
Sep 25 22:58:56 dist-test-slave-x1lh krb5kdc[10171](info): commencing operation
krb5kdc: starting...
W20260925 22:58:57.909318 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:58:57 dist-test-slave-x1lh krb5kdc[10171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:58:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:58:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:58:58.040270 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:45675
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:33033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:45675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:58.154348 10184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:58.154686 10184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:58.154765 10184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:58.158879 10184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:58:58.158996 10184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:58.159037 10184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:58.159091 10184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:58:58.159129 10184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:58:58.164661 10184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:33033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:45675
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:45675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:58.165899 10184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:58.167074 10184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:58:58.176066 10189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:58.176182 10184 server_base.cc:1034] running on GCE node
W20260925 22:58:58.176281 10192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:58.176066 10190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:58.177109 10184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:58.178234 10184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:58.179427 10184 hybrid_clock.cc:648] HybridClock initialized: now 1790377138179400 us; error 36 us; skew 500 ppm
Sep 25 22:58:58 dist-test-slave-x1lh krb5kdc[10171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377138, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:58.194135 10184 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:58:58.195240 10184 webserver.cc:460] Webserver started at http://127.20.114.254:32929/ using document root <none> and password file <none>
I20260925 22:58:58.195811 10184 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:58.195873 10184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:58.196074 10184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:58.198271 10184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "20daf49f08494fd9b712f14e1989e2d3"
format_stamp: "Formatted at 2026-09-25 22:58:58 on dist-test-slave-x1lh"
server_key: "8cf46759f69963961c136dd03e17e4d5"
server_key_iv: "8f55c7e727b43266d11c98dbe547ee58"
server_key_version: "encryptionkey@0"
I20260925 22:58:58.198837 10184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "20daf49f08494fd9b712f14e1989e2d3"
format_stamp: "Formatted at 2026-09-25 22:58:58 on dist-test-slave-x1lh"
server_key: "8cf46759f69963961c136dd03e17e4d5"
server_key_iv: "8f55c7e727b43266d11c98dbe547ee58"
server_key_version: "encryptionkey@0"
I20260925 22:58:58.202863 10184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 22:58:58.205539 10199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:58.206934 10184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:58:58.207094 10184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "20daf49f08494fd9b712f14e1989e2d3"
format_stamp: "Formatted at 2026-09-25 22:58:58 on dist-test-slave-x1lh"
server_key: "8cf46759f69963961c136dd03e17e4d5"
server_key_iv: "8f55c7e727b43266d11c98dbe547ee58"
server_key_version: "encryptionkey@0"
I20260925 22:58:58.207226 10184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:58.222900 10184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:58.226513 10184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:58.226742 10184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:58.234534 10184 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:45675
I20260925 22:58:58.234561 10251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:45675 every 8 connection(s)
I20260925 22:58:58.235612 10184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:58:58.237536 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 10184
I20260925 22:58:58.237682 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:58:58.237989 20939 external_mini_cluster.cc:1428] Setting key a6de4d73dcb349bc363947fa143dceff
I20260925 22:58:58.239338 10252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:58.245775 10252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20daf49f08494fd9b712f14e1989e2d3: Bootstrap starting.
I20260925 22:58:58.247700 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:58.239643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53763 (local address 127.20.114.254:45675)
0925 22:58:58.240232 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:58.240249 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:58.240288 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:58:58.241109 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:58.241130 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:58.241505 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:58.241869 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:58.242308 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:58.243321 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:58.244249 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:58.244747 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:58.244958 (+ 211us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:58.245965 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:58.245973 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:58.245990 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:58.246408 (+ 418us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:58.246468 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:58.246491 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:58.246630 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:58.246812 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 22:58:58.246961 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":246,"threads_started":1}
I20260925 22:58:58.248484 10252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20daf49f08494fd9b712f14e1989e2d3: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:58.249476 10252 log.cc:824] T 00000000000000000000000000000000 P 20daf49f08494fd9b712f14e1989e2d3: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:58.251528 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:58:58.252049 10252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20daf49f08494fd9b712f14e1989e2d3: No bootstrap required, opened a new log
I20260925 22:58:58.257216 10252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20daf49f08494fd9b712f14e1989e2d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20daf49f08494fd9b712f14e1989e2d3" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45675 } }
I20260925 22:58:58.257589 10252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20daf49f08494fd9b712f14e1989e2d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:58.257665 10252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20daf49f08494fd9b712f14e1989e2d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20daf49f08494fd9b712f14e1989e2d3, State: Initialized, Role: FOLLOWER
I20260925 22:58:58.258255 10252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20daf49f08494fd9b712f14e1989e2d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20daf49f08494fd9b712f14e1989e2d3" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45675 } }
I20260925 22:58:58.258427 10252 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20daf49f08494fd9b712f14e1989e2d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:58:58.258523 10252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20daf49f08494fd9b712f14e1989e2d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:58:58.258622 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:58.253744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38163 (local address 127.20.114.254:45675)
0925 22:58:58.253893 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:58.253899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:58.254075 (+ 176us) server_negotiation.cc:407] Connection header received
0925 22:58:58.254219 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:58.254221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:58.254263 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:58.254347 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:58.254729 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:58.255232 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:58.256549 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:58:58.256781 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:58:58.256949 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:58:58.258042 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:58.258046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:58.258048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:58.258062 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:58.258089 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:58.258100 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:58.258203 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:58.258360 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 22:58:58.258404 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:58:58.258649 10252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20daf49f08494fd9b712f14e1989e2d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:58.259836 10252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20daf49f08494fd9b712f14e1989e2d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20daf49f08494fd9b712f14e1989e2d3" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45675 } }
I20260925 22:58:58.260238 10252 leader_election.cc:302] T 00000000000000000000000000000000 P 20daf49f08494fd9b712f14e1989e2d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20daf49f08494fd9b712f14e1989e2d3; no voters:
I20260925 22:58:58.260609 10252 leader_election.cc:288] T 00000000000000000000000000000000 P 20daf49f08494fd9b712f14e1989e2d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:58:58.260761 10262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20daf49f08494fd9b712f14e1989e2d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:58.261159 10262 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20daf49f08494fd9b712f14e1989e2d3 [term 1 LEADER]: Becoming Leader. State: Replica: 20daf49f08494fd9b712f14e1989e2d3, State: Running, Role: LEADER
I20260925 22:58:58.261535 10262 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20daf49f08494fd9b712f14e1989e2d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20daf49f08494fd9b712f14e1989e2d3" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45675 } }
I20260925 22:58:58.262108 10252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20daf49f08494fd9b712f14e1989e2d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:58:58.263537 10263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20daf49f08494fd9b712f14e1989e2d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20daf49f08494fd9b712f14e1989e2d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20daf49f08494fd9b712f14e1989e2d3" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45675 } } }
I20260925 22:58:58.263725 10263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20daf49f08494fd9b712f14e1989e2d3 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:58.263533 10264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20daf49f08494fd9b712f14e1989e2d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20daf49f08494fd9b712f14e1989e2d3. Latest consensus state: current_term: 1 leader_uuid: "20daf49f08494fd9b712f14e1989e2d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20daf49f08494fd9b712f14e1989e2d3" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45675 } } }
I20260925 22:58:58.264194 10273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:58:58.264261 10264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20daf49f08494fd9b712f14e1989e2d3 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:58.268244 10273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:58:58.275159 10273 catalog_manager.cc:1347] Generated new cluster ID: db81fd2b3ef64c62bcae7f297dffeaa0
I20260925 22:58:58.275252 10273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:58:58.283800 10273 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:58:58.285065 10273 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:58:58.291121 10273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20daf49f08494fd9b712f14e1989e2d3: Generated new TSK 0
I20260925 22:58:58.292222 10273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:58:58.298115 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 10184
W20260925 22:58:58.304173 10169 connection.cc:535] client connection to 127.20.114.254:45675 recv error: Network error: failed to read from TLS socket (remote: 127.20.114.254:45675): Connection reset by peer (error 104)
2026-09-25T22:58:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1490 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:58:58 dist-test-slave-x1lh krb5kdc[10283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:58:58 dist-test-slave-x1lh krb5kdc[10283](info): set up 2 sockets
Sep 25 22:58:58 dist-test-slave-x1lh krb5kdc[10283](info): commencing operation
krb5kdc: starting...
W20260925 22:58:59.389302 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:58:59 dist-test-slave-x1lh krb5kdc[10283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:58:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:58:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:58:59.530210 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:33027
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:46239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:33027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:58:59.648707 10296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:58:59.649163 10296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:58:59.649284 10296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:58:59.654083 10296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:58:59.654237 10296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:58:59.654279 10296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:58:59.654325 10296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:58:59.654366 10296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:58:59.660796 10296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:46239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:33027
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:33027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:58:59.662178 10296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:58:59.663383 10296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:58:59.673632 10296 server_base.cc:1034] running on GCE node
W20260925 22:58:59.673568 10302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:59.673574 10304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:58:59.673556 10301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:58:59.674588 10296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:58:59.675875 10296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:58:59.677067 10296 hybrid_clock.cc:648] HybridClock initialized: now 1790377139677046 us; error 37 us; skew 500 ppm
Sep 25 22:58:59 dist-test-slave-x1lh krb5kdc[10283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377139, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:58:59.680672 10296 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:58:59.681795 10296 webserver.cc:460] Webserver started at http://127.20.114.254:38301/ using document root <none> and password file <none>
I20260925 22:58:59.682964 10296 fs_manager.cc:359] Metadata directory not provided
I20260925 22:58:59.683108 10296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:58:59.683393 10296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:58:59.685842 10296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "0e6935e4d785410f96980440159e6042"
format_stamp: "Formatted at 2026-09-25 22:58:59 on dist-test-slave-x1lh"
server_key: "28e28f487786c37ced40a96d3e0b38aa"
server_key_iv: "2b10c07a80f43ebf964e4aaab67370dc"
server_key_version: "encryptionkey@0"
I20260925 22:58:59.686498 10296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "0e6935e4d785410f96980440159e6042"
format_stamp: "Formatted at 2026-09-25 22:58:59 on dist-test-slave-x1lh"
server_key: "28e28f487786c37ced40a96d3e0b38aa"
server_key_iv: "2b10c07a80f43ebf964e4aaab67370dc"
server_key_version: "encryptionkey@0"
I20260925 22:58:59.690797 10296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:58:59.693897 10311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:58:59.695322 10296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:58:59.695484 10296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "0e6935e4d785410f96980440159e6042"
format_stamp: "Formatted at 2026-09-25 22:58:59 on dist-test-slave-x1lh"
server_key: "28e28f487786c37ced40a96d3e0b38aa"
server_key_iv: "2b10c07a80f43ebf964e4aaab67370dc"
server_key_version: "encryptionkey@0"
I20260925 22:58:59.695622 10296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:58:59.700627 10296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:58:59.704313 10296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:58:59.704546 10296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:58:59.712393 10296 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:33027
I20260925 22:58:59.712416 10363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:33027 every 8 connection(s)
I20260925 22:58:59.713462 10296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:58:59.716471 10364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:58:59.717460 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 10296
I20260925 22:58:59.717623 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:58:59.717972 20939 external_mini_cluster.cc:1428] Setting key 02c8a5625dace956c76a834714211280
I20260925 22:58:59.723603 10364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e6935e4d785410f96980440159e6042: Bootstrap starting.
I20260925 22:58:59.723832 10367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:59.719423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43011 (local address 127.20.114.254:33027)
0925 22:58:59.719938 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:59.719955 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:59.719992 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:58:59.720693 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:59.720713 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:59.721100 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:59.721410 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:59.722329 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:59.722336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:59.722353 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:59.722364 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:59.722426 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:59.722445 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:59.722565 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:59.722730 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 22:58:59.723031 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":172,"threads_started":1}
I20260925 22:58:59.726466 10364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e6935e4d785410f96980440159e6042: Neither blocks nor log segments found. Creating new log.
I20260925 22:58:59.727201 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:58:59.727378 10364 log.cc:824] T 00000000000000000000000000000000 P 0e6935e4d785410f96980440159e6042: Log is configured to *not* fsync() on all Append() calls
I20260925 22:58:59.729717 10364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e6935e4d785410f96980440159e6042: No bootstrap required, opened a new log
I20260925 22:58:59.731350 10367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:58:59.729442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33601 (local address 127.20.114.254:33027)
0925 22:58:59.729577 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:58:59.729582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:58:59.729897 (+ 315us) server_negotiation.cc:407] Connection header received
0925 22:58:59.729956 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:58:59.729959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:58:59.729996 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:58:59.730089 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:58:59.730871 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:58:59.730903 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:58:59.730905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:58:59.730917 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:58:59.730944 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:58:59.730955 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:58:59.731035 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 22:58:59.731148 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 22:58:59.731184 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:58:59.734081 10364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e6935e4d785410f96980440159e6042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6935e4d785410f96980440159e6042" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 33027 } }
I20260925 22:58:59.734339 10364 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e6935e4d785410f96980440159e6042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:58:59.734376 10364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e6935e4d785410f96980440159e6042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e6935e4d785410f96980440159e6042, State: Initialized, Role: FOLLOWER
I20260925 22:58:59.734812 10364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e6935e4d785410f96980440159e6042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6935e4d785410f96980440159e6042" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 33027 } }
I20260925 22:58:59.734934 10364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e6935e4d785410f96980440159e6042 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:58:59.734977 10364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e6935e4d785410f96980440159e6042 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:58:59.735050 10364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e6935e4d785410f96980440159e6042 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:58:59.735983 10364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e6935e4d785410f96980440159e6042 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6935e4d785410f96980440159e6042" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 33027 } }
I20260925 22:58:59.736299 10364 leader_election.cc:302] T 00000000000000000000000000000000 P 0e6935e4d785410f96980440159e6042 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e6935e4d785410f96980440159e6042; no voters:
I20260925 22:58:59.736579 10364 leader_election.cc:288] T 00000000000000000000000000000000 P 0e6935e4d785410f96980440159e6042 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:58:59.736766 10374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e6935e4d785410f96980440159e6042 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:58:59.737066 10374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e6935e4d785410f96980440159e6042 [term 1 LEADER]: Becoming Leader. State: Replica: 0e6935e4d785410f96980440159e6042, State: Running, Role: LEADER
I20260925 22:58:59.737391 10374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e6935e4d785410f96980440159e6042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6935e4d785410f96980440159e6042" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 33027 } }
I20260925 22:58:59.737924 10364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e6935e4d785410f96980440159e6042 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:58:59.739274 10375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e6935e4d785410f96980440159e6042 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e6935e4d785410f96980440159e6042" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6935e4d785410f96980440159e6042" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 33027 } } }
I20260925 22:58:59.739447 10375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e6935e4d785410f96980440159e6042 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:59.739939 10376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e6935e4d785410f96980440159e6042 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e6935e4d785410f96980440159e6042. Latest consensus state: current_term: 1 leader_uuid: "0e6935e4d785410f96980440159e6042" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6935e4d785410f96980440159e6042" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 33027 } } }
I20260925 22:58:59.740118 10376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e6935e4d785410f96980440159e6042 [sys.catalog]: This master's current role is: LEADER
I20260925 22:58:59.740391 10383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:58:59.744225 10383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:58:59.750221 10383 catalog_manager.cc:1347] Generated new cluster ID: 3c356fe00a3143649772c60cca2658fb
I20260925 22:58:59.750371 10383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:58:59.760562 10383 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:58:59.760991 10383 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:58:59.770993 10383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e6935e4d785410f96980440159e6042: Generated new TSK 0
I20260925 22:58:59.771838 10383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:58:59.781975 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 10296
2026-09-25T22:58:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1483 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:58:59 dist-test-slave-x1lh krb5kdc[10395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:58:59 dist-test-slave-x1lh krb5kdc[10395](info): set up 2 sockets
Sep 25 22:58:59 dist-test-slave-x1lh krb5kdc[10395](info): commencing operation
krb5kdc: starting...
W20260925 22:59:00.873804 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:59:00 dist-test-slave-x1lh krb5kdc[10395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:59:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:59:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:59:01.005187 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:35057
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:35057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:59:01.127426 10408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:59:01.127738 10408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:59:01.127782 10408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:59:01.132364 10408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:59:01.132440 10408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:59:01.132462 10408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:59:01.132479 10408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:59:01.132539 10408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:59:01.138357 10408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:35057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:35057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:59:01.139551 10408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:59:01.140631 10408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:59:01.149686 10416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:59:01.149933 10408 server_base.cc:1034] running on GCE node
W20260925 22:59:01.149688 10413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:59:01.149672 10414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:59:01.150697 10408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:59:01.151808 10408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:59:01.152951 10408 hybrid_clock.cc:648] HybridClock initialized: now 1790377141152929 us; error 33 us; skew 500 ppm
Sep 25 22:59:01 dist-test-slave-x1lh krb5kdc[10395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377141, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:59:01.155642 10408 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:59:01.156512 10408 webserver.cc:460] Webserver started at http://127.20.114.254:45865/ using document root <none> and password file <none>
I20260925 22:59:01.157130 10408 fs_manager.cc:359] Metadata directory not provided
I20260925 22:59:01.157196 10408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:59:01.157369 10408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:59:01.159488 10408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "300a1cbdca4c4541943b07366e44554f"
format_stamp: "Formatted at 2026-09-25 22:59:01 on dist-test-slave-x1lh"
server_key: "c8290886d1cd3b56d69ba196ede0589d"
server_key_iv: "42d6f69b00d0c08f39cf29603daeddb1"
server_key_version: "encryptionkey@0"
I20260925 22:59:01.160029 10408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "300a1cbdca4c4541943b07366e44554f"
format_stamp: "Formatted at 2026-09-25 22:59:01 on dist-test-slave-x1lh"
server_key: "c8290886d1cd3b56d69ba196ede0589d"
server_key_iv: "42d6f69b00d0c08f39cf29603daeddb1"
server_key_version: "encryptionkey@0"
I20260925 22:59:01.163954 10408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 22:59:01.166625 10423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:59:01.168004 10408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:59:01.168120 10408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "300a1cbdca4c4541943b07366e44554f"
format_stamp: "Formatted at 2026-09-25 22:59:01 on dist-test-slave-x1lh"
server_key: "c8290886d1cd3b56d69ba196ede0589d"
server_key_iv: "42d6f69b00d0c08f39cf29603daeddb1"
server_key_version: "encryptionkey@0"
I20260925 22:59:01.168222 10408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:59:01.195088 10408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:59:01.199018 10408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:59:01.199198 10408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:59:01.207218 10408 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:35057
I20260925 22:59:01.207238 10475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:35057 every 8 connection(s)
I20260925 22:59:01.208285 10408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:59:01.211272 10476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:59:01.213008 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 10408
I20260925 22:59:01.213166 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:59:01.213490 20939 external_mini_cluster.cc:1428] Setting key e20322acfbe7117cfcb18bbcc7ca72b7
I20260925 22:59:01.218549 10476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 300a1cbdca4c4541943b07366e44554f: Bootstrap starting.
I20260925 22:59:01.221517 10476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 300a1cbdca4c4541943b07366e44554f: Neither blocks nor log segments found. Creating new log.
I20260925 22:59:01.222558 10476 log.cc:824] T 00000000000000000000000000000000 P 300a1cbdca4c4541943b07366e44554f: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:59:01 dist-test-slave-x1lh krb5kdc[10395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:59:01.225323 10476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 300a1cbdca4c4541943b07366e44554f: No bootstrap required, opened a new log
I20260925 22:59:01.229738 10476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 300a1cbdca4c4541943b07366e44554f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "300a1cbdca4c4541943b07366e44554f" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35057 } }
I20260925 22:59:01.230031 10476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 300a1cbdca4c4541943b07366e44554f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:59:01.230118 10476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 300a1cbdca4c4541943b07366e44554f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 300a1cbdca4c4541943b07366e44554f, State: Initialized, Role: FOLLOWER
I20260925 22:59:01.230121 10479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:01.215158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40613 (local address 127.20.114.254:35057)
0925 22:59:01.215675 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:01.215692 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:01.215733 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:59:01.216468 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:01.216488 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:01.216946 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:01.217273 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:01.218241 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:01.219338 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:01.220615 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:01.221525 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:01.222198 (+ 673us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:01.224963 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:01.224969 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:01.224985 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:01.225013 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:01.226817 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:01.227398 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:01.227403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:01.227408 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:01.227556 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:01.227888 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:01.227890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:01.227891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:01.228325 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:01.228512 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:01.228815 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 22:59:01.229227 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":152,"threads_started":1}
I20260925 22:59:01.230677 10476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 300a1cbdca4c4541943b07366e44554f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "300a1cbdca4c4541943b07366e44554f" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35057 } }
I20260925 22:59:01.230849 10476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 300a1cbdca4c4541943b07366e44554f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:59:01.230938 10476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 300a1cbdca4c4541943b07366e44554f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:59:01.231065 10476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 300a1cbdca4c4541943b07366e44554f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:59:01.232076 10476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 300a1cbdca4c4541943b07366e44554f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "300a1cbdca4c4541943b07366e44554f" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35057 } }
I20260925 22:59:01.232486 10476 leader_election.cc:302] T 00000000000000000000000000000000 P 300a1cbdca4c4541943b07366e44554f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 300a1cbdca4c4541943b07366e44554f; no voters:
I20260925 22:59:01.232963 10476 leader_election.cc:288] T 00000000000000000000000000000000 P 300a1cbdca4c4541943b07366e44554f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:59:01.233055 10481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 300a1cbdca4c4541943b07366e44554f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:59:01.233268 10481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 300a1cbdca4c4541943b07366e44554f [term 1 LEADER]: Becoming Leader. State: Replica: 300a1cbdca4c4541943b07366e44554f, State: Running, Role: LEADER
I20260925 22:59:01.233572 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:59:01.233672 10481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 300a1cbdca4c4541943b07366e44554f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "300a1cbdca4c4541943b07366e44554f" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35057 } }
I20260925 22:59:01.234398 10476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 300a1cbdca4c4541943b07366e44554f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:59:01.235271 10483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 300a1cbdca4c4541943b07366e44554f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 300a1cbdca4c4541943b07366e44554f. Latest consensus state: current_term: 1 leader_uuid: "300a1cbdca4c4541943b07366e44554f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "300a1cbdca4c4541943b07366e44554f" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35057 } } }
I20260925 22:59:01.235384 10483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 300a1cbdca4c4541943b07366e44554f [sys.catalog]: This master's current role is: LEADER
I20260925 22:59:01.235728 10482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 300a1cbdca4c4541943b07366e44554f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "300a1cbdca4c4541943b07366e44554f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "300a1cbdca4c4541943b07366e44554f" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35057 } } }
I20260925 22:59:01.235831 10482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 300a1cbdca4c4541943b07366e44554f [sys.catalog]: This master's current role is: LEADER
I20260925 22:59:01.236186 10494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:59:01.240137 10494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:59:01.246465 10479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:01.236448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41509 (local address 127.20.114.254:35057)
0925 22:59:01.236579 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:01.236584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:01.237172 (+ 588us) server_negotiation.cc:407] Connection header received
0925 22:59:01.237356 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:01.237360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:01.237415 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:01.237492 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:01.238379 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:01.239211 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:01.240360 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:01.240704 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:01.240815 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:01.242723 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:01.242727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:01.242730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:01.242758 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:01.244577 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:01.245167 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:01.245170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:01.245171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:01.245249 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:01.245543 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:01.245546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:01.245548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:01.245744 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:01.245879 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:01.246125 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 22:59:01.246260 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:59:01.247367 10494 catalog_manager.cc:1347] Generated new cluster ID: 8cba51304e194c04a10c3df929e1640a
I20260925 22:59:01.247459 10494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:59:01.258392 10494 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:59:01.259591 10494 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:59:01.281670 10494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 300a1cbdca4c4541943b07366e44554f: Generated new TSK 0
I20260925 22:59:01.282554 10494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:59:01.292444 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 10408
2026-09-25T22:59:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1510 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:59:01 dist-test-slave-x1lh krb5kdc[10507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:59:01 dist-test-slave-x1lh krb5kdc[10507](info): set up 2 sockets
Sep 25 22:59:01 dist-test-slave-x1lh krb5kdc[10507](info): commencing operation
krb5kdc: starting...
W20260925 22:59:02.349506 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:59:02 dist-test-slave-x1lh krb5kdc[10507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:59:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:59:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:59:02.484761 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:45009
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:42473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:45009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:59:02.604471 10520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:59:02.604796 10520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:59:02.604842 10520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:59:02.609280 10520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:59:02.609370 10520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:59:02.609390 10520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:59:02.609408 10520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:59:02.609424 10520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:59:02.615655 10520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:42473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:45009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:45009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:59:02.616844 10520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:59:02.617967 10520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:59:02.627566 10526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:59:02.627557 10528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:59:02.627619 10520 server_base.cc:1034] running on GCE node
W20260925 22:59:02.627557 10525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:59:02.628515 10520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:59:02.629817 10520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:59:02.631024 10520 hybrid_clock.cc:648] HybridClock initialized: now 1790377142630991 us; error 48 us; skew 500 ppm
Sep 25 22:59:02 dist-test-slave-x1lh krb5kdc[10507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377142, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:59:02.634227 10520 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:59:02.635268 10520 webserver.cc:460] Webserver started at http://127.20.114.254:44327/ using document root <none> and password file <none>
I20260925 22:59:02.635835 10520 fs_manager.cc:359] Metadata directory not provided
I20260925 22:59:02.635900 10520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:59:02.636096 10520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:59:02.638391 10520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "adcd10dfa6c749ed8adb5e783a064605"
format_stamp: "Formatted at 2026-09-25 22:59:02 on dist-test-slave-x1lh"
server_key: "f41561863de3bf2ffcb322092776a5dd"
server_key_iv: "04876519a2a4040a5e18a4c71fe8a515"
server_key_version: "encryptionkey@0"
I20260925 22:59:02.638962 10520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "adcd10dfa6c749ed8adb5e783a064605"
format_stamp: "Formatted at 2026-09-25 22:59:02 on dist-test-slave-x1lh"
server_key: "f41561863de3bf2ffcb322092776a5dd"
server_key_iv: "04876519a2a4040a5e18a4c71fe8a515"
server_key_version: "encryptionkey@0"
I20260925 22:59:02.643119 10520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:59:02.645834 10535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:59:02.647246 10520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:59:02.647416 10520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "adcd10dfa6c749ed8adb5e783a064605"
format_stamp: "Formatted at 2026-09-25 22:59:02 on dist-test-slave-x1lh"
server_key: "f41561863de3bf2ffcb322092776a5dd"
server_key_iv: "04876519a2a4040a5e18a4c71fe8a515"
server_key_version: "encryptionkey@0"
I20260925 22:59:02.647550 10520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:59:02.664817 10520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:59:02.668519 10520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:59:02.668746 10520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:59:02.676816 10520 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:45009
I20260925 22:59:02.676842 10587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:45009 every 8 connection(s)
I20260925 22:59:02.677835 10520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:59:02.680826 10588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:59:02.682161 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 10520
I20260925 22:59:02.682374 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:59:02.682708 20939 external_mini_cluster.cc:1428] Setting key de3f4bac17c99505d69908230d5c8ff7
I20260925 22:59:02.687726 10588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605: Bootstrap starting.
I20260925 22:59:02.690832 10588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605: Neither blocks nor log segments found. Creating new log.
I20260925 22:59:02.691672 10588 log.cc:824] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605: Log is configured to *not* fsync() on all Append() calls
I20260925 22:59:02.693215 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:02.684540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40535 (local address 127.20.114.254:45009)
0925 22:59:02.685077 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:02.685104 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:02.685143 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:59:02.685817 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:02.685836 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:02.686218 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:02.686515 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:02.686934 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:02.688068 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:02.689397 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:02.689930 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:02.690067 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:02.691105 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:02.691110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:02.691128 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:59:02.691559 (+ 431us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:02.691623 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:59:02.691648 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:02.691731 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:02.692007 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 22:59:02.692258 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":238,"threads_started":1}
I20260925 22:59:02.694108 10588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605: No bootstrap required, opened a new log
I20260925 22:59:02.696264 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:59:02.698371 10588 raft_consensus.cc:348] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcd10dfa6c749ed8adb5e783a064605" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45009 } }
I20260925 22:59:02.698761 10588 raft_consensus.cc:374] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:59:02.698844 10588 raft_consensus.cc:729] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adcd10dfa6c749ed8adb5e783a064605, State: Initialized, Role: FOLLOWER
I20260925 22:59:02.699364 10588 consensus_queue.cc:260] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcd10dfa6c749ed8adb5e783a064605" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45009 } }
I20260925 22:59:02.699548 10588 raft_consensus.cc:388] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:59:02.699647 10588 raft_consensus.cc:482] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:59:02.699787 10588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:59:02.700912 10588 raft_consensus.cc:504] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcd10dfa6c749ed8adb5e783a064605" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45009 } }
I20260925 22:59:02.701336 10588 leader_election.cc:302] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adcd10dfa6c749ed8adb5e783a064605; no voters:
I20260925 22:59:02.701769 10588 leader_election.cc:288] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:59:02.701892 10598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:59:02.702405 10598 raft_consensus.cc:686] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605 [term 1 LEADER]: Becoming Leader. State: Replica: adcd10dfa6c749ed8adb5e783a064605, State: Running, Role: LEADER
I20260925 22:59:02.702850 10598 consensus_queue.cc:237] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcd10dfa6c749ed8adb5e783a064605" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45009 } }
I20260925 22:59:02.703091 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:02.698532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53795 (local address 127.20.114.254:45009)
0925 22:59:02.698680 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:02.698685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:02.698864 (+ 179us) server_negotiation.cc:407] Connection header received
0925 22:59:02.699028 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:02.699030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:02.699073 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:02.699167 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:02.699456 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:02.700206 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:02.701178 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:02.701461 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:02.701599 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:02.702601 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:02.702604 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:02.702607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:59:02.702622 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:02.702653 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:59:02.702667 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:02.702742 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:02.702890 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 22:59:02.702957 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:59:02.703416 10588 sys_catalog.cc:564] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:59:02.705544 10600 sys_catalog.cc:455] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adcd10dfa6c749ed8adb5e783a064605. Latest consensus state: current_term: 1 leader_uuid: "adcd10dfa6c749ed8adb5e783a064605" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcd10dfa6c749ed8adb5e783a064605" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45009 } } }
I20260925 22:59:02.706036 10600 sys_catalog.cc:458] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605 [sys.catalog]: This master's current role is: LEADER
I20260925 22:59:02.706288 10599 sys_catalog.cc:455] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adcd10dfa6c749ed8adb5e783a064605" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcd10dfa6c749ed8adb5e783a064605" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 45009 } } }
I20260925 22:59:02.706393 10599 sys_catalog.cc:458] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605 [sys.catalog]: This master's current role is: LEADER
W20260925 22:59:02.708460 10612 catalog_manager.cc:1558] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:59:02.708539 10612 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:59:02.708630 10613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:59:02.709666 10613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:59:02.716531 10613 catalog_manager.cc:1347] Generated new cluster ID: 22188bf9ecdc4bfca5d736d75f81cb5d
I20260925 22:59:02.716630 10613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:59:02.729815 10613 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:59:02.730878 10613 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:59:02.741389 10613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adcd10dfa6c749ed8adb5e783a064605: Generated new TSK 0
I20260925 22:59:02.742194 10613 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:59:02.747808 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 10520
2026-09-25T22:59:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1455 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:59:02 dist-test-slave-x1lh krb5kdc[10619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:59:02 dist-test-slave-x1lh krb5kdc[10619](info): set up 2 sockets
Sep 25 22:59:02 dist-test-slave-x1lh krb5kdc[10619](info): commencing operation
krb5kdc: starting...
W20260925 22:59:03.837852 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:59:03 dist-test-slave-x1lh krb5kdc[10619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:59:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:59:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:59:03.977443 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:33911
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:33379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:33911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:59:04.099848 10632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:59:04.100178 10632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:59:04.100229 10632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:59:04.104943 10632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:59:04.105022 10632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:59:04.105043 10632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:59:04.105062 10632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:59:04.105114 10632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:59:04.111183 10632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:33379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:33911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:33911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:59:04.112435 10632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:59:04.113686 10632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:59:04.123601 10638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:59:04.123610 10632 server_base.cc:1034] running on GCE node
W20260925 22:59:04.123553 10637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:59:04.123569 10640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:59:04.124568 10632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:59:04.125864 10632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:59:04.127058 10632 hybrid_clock.cc:648] HybridClock initialized: now 1790377144127030 us; error 37 us; skew 500 ppm
Sep 25 22:59:04 dist-test-slave-x1lh krb5kdc[10619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377144, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:59:04.135114 10632 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:59:04.136341 10632 webserver.cc:460] Webserver started at http://127.20.114.254:41669/ using document root <none> and password file <none>
I20260925 22:59:04.137118 10632 fs_manager.cc:359] Metadata directory not provided
I20260925 22:59:04.137189 10632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:59:04.137421 10632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:59:04.139700 10632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "e7b70b67c59641ecb4a8a65cf4fb9166"
format_stamp: "Formatted at 2026-09-25 22:59:04 on dist-test-slave-x1lh"
server_key: "754601347f831c757b22da4324caf9e8"
server_key_iv: "192f1ab92c98355bb804dab8de471fc1"
server_key_version: "encryptionkey@0"
I20260925 22:59:04.140383 10632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "e7b70b67c59641ecb4a8a65cf4fb9166"
format_stamp: "Formatted at 2026-09-25 22:59:04 on dist-test-slave-x1lh"
server_key: "754601347f831c757b22da4324caf9e8"
server_key_iv: "192f1ab92c98355bb804dab8de471fc1"
server_key_version: "encryptionkey@0"
I20260925 22:59:04.144766 10632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 22:59:04.147864 10647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:59:04.149431 10632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 22:59:04.149591 10632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "e7b70b67c59641ecb4a8a65cf4fb9166"
format_stamp: "Formatted at 2026-09-25 22:59:04 on dist-test-slave-x1lh"
server_key: "754601347f831c757b22da4324caf9e8"
server_key_iv: "192f1ab92c98355bb804dab8de471fc1"
server_key_version: "encryptionkey@0"
I20260925 22:59:04.149804 10632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:59:04.154932 10632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:59:04.158469 10632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:59:04.158699 10632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:59:04.166194 10632 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:33911
I20260925 22:59:04.166213 10699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:33911 every 8 connection(s)
I20260925 22:59:04.167239 10632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:59:04.170328 10700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:59:04.175382 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 10632
I20260925 22:59:04.175530 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:59:04.175817 20939 external_mini_cluster.cc:1428] Setting key 5f6c2b1e55a9365f5108f0690ee0d3c2
I20260925 22:59:04.176688 10700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166: Bootstrap starting.
I20260925 22:59:04.179320 10700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166: Neither blocks nor log segments found. Creating new log.
I20260925 22:59:04.180272 10700 log.cc:824] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166: Log is configured to *not* fsync() on all Append() calls
I20260925 22:59:04.181648 10703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:04.177450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50955 (local address 127.20.114.254:33911)
0925 22:59:04.177960 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:04.177976 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:04.178014 (+ 38us) server_negotiation.cc:407] Connection header received
0925 22:59:04.178713 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:04.178731 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:04.179044 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:04.179422 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:04.180208 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:04.180212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:04.180224 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:59:04.180234 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:04.180292 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:59:04.180312 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:04.180429 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:04.180539 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 22:59:04.180731 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":161,"threads_started":1}
I20260925 22:59:04.182596 10700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166: No bootstrap required, opened a new log
I20260925 22:59:04.184763 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:59:04.186800 10700 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b70b67c59641ecb4a8a65cf4fb9166" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 33911 } }
I20260925 22:59:04.187186 10700 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:59:04.187253 10700 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7b70b67c59641ecb4a8a65cf4fb9166, State: Initialized, Role: FOLLOWER
I20260925 22:59:04.187777 10700 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b70b67c59641ecb4a8a65cf4fb9166" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 33911 } }
I20260925 22:59:04.187974 10700 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:59:04.188052 10700 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:59:04.188166 10700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:59:04.188791 10703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:04.186948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42739 (local address 127.20.114.254:33911)
0925 22:59:04.187099 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:04.187104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:04.187339 (+ 235us) server_negotiation.cc:407] Connection header received
0925 22:59:04.187456 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:04.187458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:04.187492 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:04.187588 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:04.188407 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:04.188413 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:04.188419 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:59:04.188435 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:04.188466 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:59:04.188480 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:04.188573 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:04.188634 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 22:59:04.188672 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:59:04.189476 10700 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b70b67c59641ecb4a8a65cf4fb9166" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 33911 } }
I20260925 22:59:04.189850 10700 leader_election.cc:302] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7b70b67c59641ecb4a8a65cf4fb9166; no voters:
I20260925 22:59:04.190167 10700 leader_election.cc:288] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:59:04.190394 10710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:59:04.190656 10710 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166 [term 1 LEADER]: Becoming Leader. State: Replica: e7b70b67c59641ecb4a8a65cf4fb9166, State: Running, Role: LEADER
I20260925 22:59:04.191009 10710 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b70b67c59641ecb4a8a65cf4fb9166" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 33911 } }
I20260925 22:59:04.191455 10700 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:59:04.192642 10712 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7b70b67c59641ecb4a8a65cf4fb9166. Latest consensus state: current_term: 1 leader_uuid: "e7b70b67c59641ecb4a8a65cf4fb9166" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b70b67c59641ecb4a8a65cf4fb9166" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 33911 } } }
I20260925 22:59:04.192821 10712 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166 [sys.catalog]: This master's current role is: LEADER
I20260925 22:59:04.193063 10711 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7b70b67c59641ecb4a8a65cf4fb9166" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b70b67c59641ecb4a8a65cf4fb9166" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 33911 } } }
I20260925 22:59:04.193156 10711 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166 [sys.catalog]: This master's current role is: LEADER
W20260925 22:59:04.196465 10725 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:59:04.196568 10725 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:59:04.196642 10720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:59:04.197449 10720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:59:04.203811 10720 catalog_manager.cc:1347] Generated new cluster ID: c71763c279ea487d94774213e2b06ece
I20260925 22:59:04.203912 10720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:59:04.220688 10720 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:59:04.221124 10720 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:59:04.226269 10720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7b70b67c59641ecb4a8a65cf4fb9166: Generated new TSK 0
I20260925 22:59:04.227072 10720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:59:04.236661 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 10632
2026-09-25T22:59:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1489 ms)
[----------] 18 tests from AuthTokenIssuingTest (26953 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:59:04 dist-test-slave-x1lh krb5kdc[10731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:59:04 dist-test-slave-x1lh krb5kdc[10731](info): set up 2 sockets
Sep 25 22:59:04 dist-test-slave-x1lh krb5kdc[10731](info): commencing operation
krb5kdc: starting...
W20260925 22:59:05.317394 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:59:05 dist-test-slave-x1lh krb5kdc[10731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:59:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:59:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:59:05.459316 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:40539
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:40539,127.20.114.253:36839,127.20.114.252:45489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:59:05.583961 10744 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:59:05.584313 10744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:59:05.584342 10744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:59:05.584383 10744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:59:05.588771 10744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:59:05.588894 10744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:59:05.588917 10744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:59:05.588936 10744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:59:05.588953 10744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:59:05.595938 10744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:40539,127.20.114.253:36839,127.20.114.252:45489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:40539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:59:05.597316 10744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:59:05.598412 10744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:59:05.608312 10750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:59:05.608373 10744 server_base.cc:1034] running on GCE node
W20260925 22:59:05.608304 10749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:59:05.608304 10752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:59:05.609330 10744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:59:05.610791 10744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:59:05.612035 10744 hybrid_clock.cc:648] HybridClock initialized: now 1790377145612005 us; error 50 us; skew 500 ppm
Sep 25 22:59:05 dist-test-slave-x1lh krb5kdc[10731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377145, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:59:05.616194 10744 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:59:05.617376 10744 webserver.cc:460] Webserver started at http://127.20.114.254:35345/ using document root <none> and password file <none>
I20260925 22:59:05.618005 10744 fs_manager.cc:359] Metadata directory not provided
I20260925 22:59:05.618094 10744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:59:05.618288 10744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:59:05.621025 10744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "927df2298f68408396669d12e2081d5a"
format_stamp: "Formatted at 2026-09-25 22:59:05 on dist-test-slave-x1lh"
server_key: "a131a69056a358629a819964ad736068"
server_key_iv: "c0bb31bbe46b66a5047026a99d9f81c3"
server_key_version: "encryptionkey@0"
I20260925 22:59:05.621805 10744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "927df2298f68408396669d12e2081d5a"
format_stamp: "Formatted at 2026-09-25 22:59:05 on dist-test-slave-x1lh"
server_key: "a131a69056a358629a819964ad736068"
server_key_iv: "c0bb31bbe46b66a5047026a99d9f81c3"
server_key_version: "encryptionkey@0"
I20260925 22:59:05.626314 10744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 22:59:05.629386 10759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:59:05.631004 10744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:59:05.631191 10744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "927df2298f68408396669d12e2081d5a"
format_stamp: "Formatted at 2026-09-25 22:59:05 on dist-test-slave-x1lh"
server_key: "a131a69056a358629a819964ad736068"
server_key_iv: "c0bb31bbe46b66a5047026a99d9f81c3"
server_key_version: "encryptionkey@0"
I20260925 22:59:05.631337 10744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:59:05.645908 10744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:59:05.649636 10744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:59:05.649906 10744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:59:05.657866 10744 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:40539
I20260925 22:59:05.658161 10811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:40539 every 8 connection(s)
I20260925 22:59:05.659116 10744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:59:05.662242 10812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:59:05.666468 10812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40539 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:59:05.667820 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 10744
I20260925 22:59:05.667960 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:59:05.668241 20939 external_mini_cluster.cc:1428] Setting key 8b1b8cba7c897248b0abb34e87594a42
WARNING: no policy specified for kudu/127.20.114.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.253@KRBTEST.COM" created.
Sep 25 22:59:05 dist-test-slave-x1lh krb5kdc[10731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377145, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:59:05.686398 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:05.670499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.20.114.254:40539)
0925 22:59:05.670862 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:05.670872 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:05.670912 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:59:05.671577 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:05.671582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:05.671789 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:05.671994 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:05.673029 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:05.674003 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:05.675107 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:05.675526 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:05.675810 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:05.679961 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:05.679969 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:05.679983 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:05.680009 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:05.682155 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:05.683182 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:05.683188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:05.683189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:05.683372 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:05.683846 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:05.683850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:05.683852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:05.684289 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:05.684415 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:05.685155 (+ 740us) server_negotiation.cc:299] Negotiation successful
0925 22:59:05.685449 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":231,"thread_start_us":109,"threads_started":1}
I20260925 22:59:05.686401 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:05.669402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:40539 (local address 127.0.0.1:37794)
0925 22:59:05.670075 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:05.670140 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 22:59:05.671218 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:59:05.671988 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:59:05.671994 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:59:05.672512 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:59:05.672782 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:05.672799 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:05.674147 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:05.674151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:05.674976 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:05.674985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:05.675698 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:05.675703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:05.675917 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:05.676544 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:59:05.676562 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:59:05.679759 (+ 3197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:59:05.682468 (+ 2709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:05.682490 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:05.682508 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:05.683004 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:05.683498 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:05.683502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:05.683505 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:05.683701 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:05.684442 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:59:05.684453 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:59:05.684712 (+ 259us) client_negotiation.cc:769] Sending connection context
0925 22:59:05.685046 (+ 334us) client_negotiation.cc:240] Negotiation successful
0925 22:59:05.685575 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":110,"thread_start_us":194,"threads_started":1}
I20260925 22:59:05.689468 10812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:59:05.690483 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:05.690165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:36839 (local address 127.0.0.1:40932)
0925 22:59:05.690319 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:05.690410 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.253:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 22:59:05.690825 10762 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.114.253:36839: connect: Connection refused (error 111)
I20260925 22:59:05.691989 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:05.691722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:36839 (local address 127.0.0.1:40948)
0925 22:59:05.691869 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:05.691920 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.253:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 22:59:05.692140 10812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.253:36839: Network error: Client connection negotiation failed: client connection to 127.20.114.253:36839: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.253.keytab.
Entry for principal kudu/127.20.114.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.253.keytab.
I20260925 22:59:05.698148 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.253:36839
--webserver_interface=127.20.114.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.253.keytab
--principal=kudu/127.20.114.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:40539,127.20.114.253:36839,127.20.114.252:45489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:59:05.741410 10812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } attempt: 1
I20260925 22:59:05.742344 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:05.742029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:36839 (local address 127.0.0.1:40956)
0925 22:59:05.742181 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:05.742262 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.253:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 22:59:05.743237 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:05.742999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:36839 (local address 127.0.0.1:40970)
0925 22:59:05.743114 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:05.743173 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.253:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 22:59:05.743407 10812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.253:36839: Network error: Client connection negotiation failed: client connection to 127.20.114.253:36839: connect: Connection refused (error 111)
I20260925 22:59:05.811619 10812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } attempt: 2
I20260925 22:59:05.812575 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:05.812243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:36839 (local address 127.0.0.1:40986)
0925 22:59:05.812415 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:05.812493 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.253:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 22:59:05.813982 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:05.813305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:36839 (local address 127.0.0.1:41002)
0925 22:59:05.813808 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:05.813901 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.253:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194}
W20260925 22:59:05.814385 10812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.253:36839: Network error: Client connection negotiation failed: client connection to 127.20.114.253:36839: connect: Connection refused (error 111)
W20260925 22:59:05.829144 10819 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:59:05.830019 10819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:59:05.830081 10819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:59:05.830266 10819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:59:05.836040 10819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:59:05.836149 10819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:59:05.836189 10819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:59:05.836225 10819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:59:05.836520 10819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:59:05.844767 10819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:40539,127.20.114.253:36839,127.20.114.252:45489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.253.keytab
--principal=kudu/127.20.114.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.253:36839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.114.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:59:05.846534 10819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:59:05.848174 10819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:59:05.859740 10827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:59:05.860086 10824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:59:05.860102 10825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:59:05.860395 10819 server_base.cc:1034] running on GCE node
I20260925 22:59:05.861208 10819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:59:05.862459 10819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:59:05.866004 10819 hybrid_clock.cc:648] HybridClock initialized: now 1790377145865978 us; error 159 us; skew 500 ppm
Sep 25 22:59:05 dist-test-slave-x1lh krb5kdc[10731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377145, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:59:05.871306 10819 init.cc:377] Logged in from keytab as kudu/127.20.114.253@KRBTEST.COM (short username kudu)
I20260925 22:59:05.872473 10819 webserver.cc:460] Webserver started at http://127.20.114.253:45471/ using document root <none> and password file <none>
I20260925 22:59:05.873186 10819 fs_manager.cc:359] Metadata directory not provided
I20260925 22:59:05.873270 10819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:59:05.873461 10819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:59:05.875780 10819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-1/data/instance:
uuid: "1051f8735870457e932c20b1c9ea9fc8"
format_stamp: "Formatted at 2026-09-25 22:59:05 on dist-test-slave-x1lh"
server_key: "bc5f16df1c4853bb5ce1bfeab196472e"
server_key_iv: "de5e3fdfbf216533e23f33ddc6c1dbd0"
server_key_version: "encryptionkey@0"
I20260925 22:59:05.876437 10819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-1/wal/instance:
uuid: "1051f8735870457e932c20b1c9ea9fc8"
format_stamp: "Formatted at 2026-09-25 22:59:05 on dist-test-slave-x1lh"
server_key: "bc5f16df1c4853bb5ce1bfeab196472e"
server_key_iv: "de5e3fdfbf216533e23f33ddc6c1dbd0"
server_key_version: "encryptionkey@0"
I20260925 22:59:05.881086 10819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260925 22:59:05.885385 10834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:59:05.887626 10819 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:59:05.887933 10819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-1/wal
uuid: "1051f8735870457e932c20b1c9ea9fc8"
format_stamp: "Formatted at 2026-09-25 22:59:05 on dist-test-slave-x1lh"
server_key: "bc5f16df1c4853bb5ce1bfeab196472e"
server_key_iv: "de5e3fdfbf216533e23f33ddc6c1dbd0"
server_key_version: "encryptionkey@0"
I20260925 22:59:05.888123 10819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:59:05.905828 10819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:59:05.905987 10812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } attempt: 3
I20260925 22:59:05.907078 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:05.906616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:36839 (local address 127.0.0.1:41018)
0925 22:59:05.906883 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:05.906973 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.253:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 22:59:05.907953 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:05.907688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:36839 (local address 127.0.0.1:41020)
0925 22:59:05.907822 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:05.907889 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.253:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 22:59:05.908108 10812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.253:36839: Network error: Client connection negotiation failed: client connection to 127.20.114.253:36839: connect: Connection refused (error 111)
I20260925 22:59:05.910287 10819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:59:05.910522 10819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:59:05.918257 10819 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.253:36839
I20260925 22:59:05.918603 10886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.253:36839 every 8 connection(s)
I20260925 22:59:05.919427 10819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-1/data/info.pb
I20260925 22:59:05.922719 10887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:59:05.926996 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 10819
I20260925 22:59:05.927132 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-1/wal/instance
I20260925 22:59:05.927459 20939 external_mini_cluster.cc:1428] Setting key 96753cf53662799176cb95c09bbc6d04
I20260925 22:59:05.927173 10887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40539 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.114.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.252@KRBTEST.COM" created.
Sep 25 22:59:05 dist-test-slave-x1lh krb5kdc[10731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377145, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.253@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:59:05.945825 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:05.930081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37810 (local address 127.20.114.254:40539)
0925 22:59:05.930224 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:05.930228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:05.931867 (+ 1639us) server_negotiation.cc:407] Connection header received
0925 22:59:05.933032 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:05.933039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:05.933115 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:05.933229 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:05.934924 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:05.935851 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:05.937507 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:05.937795 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:05.937894 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:05.941213 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:05.941225 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:05.941227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:05.941258 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:05.943055 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:05.943701 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:05.943708 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:05.943713 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:05.943819 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:05.944211 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:05.944217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:05.944222 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:05.944525 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:05.944645 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:05.945368 (+ 723us) server_negotiation.cc:299] Negotiation successful
0925 22:59:05.945552 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:59:05.946484 10890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:05.930725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:40539 (local address 127.0.0.1:37810)
0925 22:59:05.931667 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:05.931726 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:59:05.932735 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:59:05.933442 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:59:05.933450 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:59:05.933845 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:59:05.934618 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:05.934640 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:05.936046 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:05.936049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:05.937363 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:05.937377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:05.938024 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:05.938031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:05.938326 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:05.939078 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:59:05.939106 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:59:05.940982 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:59:05.943215 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:05.943223 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:05.943243 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:05.943566 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:05.943926 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:05.943929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:05.943931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:05.944090 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:05.944666 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:59:05.944671 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:59:05.945055 (+ 384us) client_negotiation.cc:769] Sending connection context
0925 22:59:05.945233 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 22:59:05.945549 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":283,"threads_started":1}
I20260925 22:59:05.948210 10887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:59:05 dist-test-slave-x1lh krb5kdc[10731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377145, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.253@KRBTEST.COM for kudu/127.20.114.253@KRBTEST.COM
I20260925 22:59:05.959331 10890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:05.948933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:36839 (local address 127.0.0.1:41036)
0925 22:59:05.949064 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:05.949086 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 22:59:05.949186 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:59:05.949733 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:59:05.949736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:59:05.950155 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:59:05.950260 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:05.950267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:05.951607 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:05.951614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:05.952241 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:05.952251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:05.952933 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:05.952937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:05.953027 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:05.953641 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:59:05.953654 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:59:05.955349 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:59:05.957709 (+ 2360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:05.957714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:05.957717 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:05.957910 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:05.958232 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:05.958234 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:05.958235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:05.958303 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:05.958782 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:59:05.958785 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:59:05.958880 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 22:59:05.958981 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 22:59:05.959131 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 22:59:05.959355 10892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:05.949022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41036 (local address 127.20.114.253:36839)
0925 22:59:05.949357 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:05.949367 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:05.949387 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:59:05.949420 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:05.949425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:05.949601 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:05.949847 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:05.950475 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:05.951400 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:05.952369 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:05.952783 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:05.952982 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:05.955519 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:05.955531 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:05.955546 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:05.955567 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:05.957585 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:05.958025 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:05.958036 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:05.958037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:05.958108 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:05.958389 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:05.958394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:05.958399 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:05.958670 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:05.958814 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:05.959088 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 22:59:05.959220 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.252.keytab.
Entry for principal kudu/127.20.114.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.252.keytab.
I20260925 22:59:05.960902 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.252:45489
--webserver_interface=127.20.114.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.252.keytab
--principal=kudu/127.20.114.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:40539,127.20.114.253:36839,127.20.114.252:45489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:59:05.961774 10887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:59:05.962584 10890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:05.962293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:45489 (local address 127.0.0.1:36046)
0925 22:59:05.962424 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:05.962512 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:45489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 22:59:05.962883 10838 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.114.252:45489: connect: Connection refused (error 111)
I20260925 22:59:05.963850 10890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:05.963643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:45489 (local address 127.0.0.1:36058)
0925 22:59:05.963751 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:05.963796 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:45489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 22:59:05.964015 10887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.252:45489: Network error: Client connection negotiation failed: client connection to 127.20.114.252:45489: connect: Connection refused (error 111)
I20260925 22:59:06.013338 10887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } attempt: 1
I20260925 22:59:06.014436 10890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.014073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:45489 (local address 127.0.0.1:36060)
0925 22:59:06.014255 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:06.014337 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:45489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 22:59:06.015316 10890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.015083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:45489 (local address 127.0.0.1:36064)
0925 22:59:06.015194 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:06.015249 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:45489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 22:59:06.015483 10887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.252:45489: Network error: Client connection negotiation failed: client connection to 127.20.114.252:45489: connect: Connection refused (error 111)
I20260925 22:59:06.051301 10812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } attempt: 4
Sep 25 22:59:06 dist-test-slave-x1lh krb5kdc[10731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377145, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for kudu/127.20.114.253@KRBTEST.COM
I20260925 22:59:06.062543 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.051930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:36839 (local address 127.0.0.1:41038)
0925 22:59:06.052208 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:06.052224 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:59:06.052325 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:59:06.052653 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:59:06.052656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:59:06.052973 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:59:06.053093 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:06.053099 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.054336 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.054342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:06.055653 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:06.055668 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.056362 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.056366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:06.056466 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:06.057080 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:59:06.057094 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:59:06.058572 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:59:06.060942 (+ 2370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:06.060945 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:06.060947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:06.061172 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:06.061493 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:06.061495 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:06.061496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:06.061574 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:06.062054 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:59:06.062056 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:59:06.062153 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 22:59:06.062241 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 22:59:06.062382 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260925 22:59:06.062650 10892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.052118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41038 (local address 127.20.114.253:36839)
0925 22:59:06.052313 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.052320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.052340 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:59:06.052426 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.052429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.052483 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.052579 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.053239 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.054160 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.055809 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.056199 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.056346 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:06.058727 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:06.058732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:06.058734 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:06.058769 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:06.060671 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.061289 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.061293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.061295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.061392 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.061674 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.061679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.061681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.061927 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:06.062032 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:06.062345 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 22:59:06.062481 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 22:59:06.063655 10812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:59:06.064498 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.064220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:45489 (local address 127.0.0.1:36078)
0925 22:59:06.064361 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:06.064415 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:45489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 22:59:06.065357 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.065068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:45489 (local address 127.0.0.1:36080)
0925 22:59:06.065182 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:06.065291 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:45489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 22:59:06.065497 10812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.252:45489: Network error: Client connection negotiation failed: client connection to 127.20.114.252:45489: connect: Connection refused (error 111)
I20260925 22:59:06.083694 10887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } attempt: 2
I20260925 22:59:06.084592 10890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.084259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:45489 (local address 127.0.0.1:36094)
0925 22:59:06.084426 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:06.084501 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:45489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 22:59:06.085500 10890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.085259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:45489 (local address 127.0.0.1:36110)
0925 22:59:06.085376 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:06.085434 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:45489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 22:59:06.085704 10887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.252:45489: Network error: Client connection negotiation failed: client connection to 127.20.114.252:45489: connect: Connection refused (error 111)
W20260925 22:59:06.089262 10893 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:59:06.089576 10893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:59:06.089648 10893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:59:06.089768 10893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:59:06.094445 10893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:59:06.094511 10893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:59:06.094530 10893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:59:06.094568 10893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:59:06.094599 10893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:59:06.100459 10893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:40539,127.20.114.253:36839,127.20.114.252:45489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.252.keytab
--principal=kudu/127.20.114.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.252:45489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.114.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:59:06.101747 10893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:59:06.102865 10893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:59:06.112010 10902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:59:06.112068 10899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:59:06.112126 10893 server_base.cc:1034] running on GCE node
W20260925 22:59:06.112030 10900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:59:06.112967 10893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:59:06.114231 10893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:59:06.115449 10893 hybrid_clock.cc:648] HybridClock initialized: now 1790377146115414 us; error 50 us; skew 500 ppm
Sep 25 22:59:06 dist-test-slave-x1lh krb5kdc[10731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377146, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:59:06.118866 10893 init.cc:377] Logged in from keytab as kudu/127.20.114.252@KRBTEST.COM (short username kudu)
I20260925 22:59:06.119928 10893 webserver.cc:460] Webserver started at http://127.20.114.252:36305/ using document root <none> and password file <none>
I20260925 22:59:06.120500 10893 fs_manager.cc:359] Metadata directory not provided
I20260925 22:59:06.120564 10893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:59:06.120738 10893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:59:06.123057 10893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-2/data/instance:
uuid: "8e62d31190a3434587c48dab2d328f7a"
format_stamp: "Formatted at 2026-09-25 22:59:06 on dist-test-slave-x1lh"
server_key: "a05b9d2a94039267ba301dded475be6f"
server_key_iv: "b38aef5f14b06ef15e82c3e0d3ca7286"
server_key_version: "encryptionkey@0"
I20260925 22:59:06.123656 10893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-2/wal/instance:
uuid: "8e62d31190a3434587c48dab2d328f7a"
format_stamp: "Formatted at 2026-09-25 22:59:06 on dist-test-slave-x1lh"
server_key: "a05b9d2a94039267ba301dded475be6f"
server_key_iv: "b38aef5f14b06ef15e82c3e0d3ca7286"
server_key_version: "encryptionkey@0"
I20260925 22:59:06.124696 10812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } attempt: 1
I20260925 22:59:06.125572 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.125267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:45489 (local address 127.0.0.1:36112)
0925 22:59:06.125430 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:06.125498 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:45489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 22:59:06.126425 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.126165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:45489 (local address 127.0.0.1:36128)
0925 22:59:06.126310 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:06.126362 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:45489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 22:59:06.126811 10812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.252:45489: Network error: Client connection negotiation failed: client connection to 127.20.114.252:45489: connect: Connection refused (error 111)
I20260925 22:59:06.128376 10893 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:59:06.131424 10909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:59:06.132756 10893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 22:59:06.133013 10893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-2/wal
uuid: "8e62d31190a3434587c48dab2d328f7a"
format_stamp: "Formatted at 2026-09-25 22:59:06 on dist-test-slave-x1lh"
server_key: "a05b9d2a94039267ba301dded475be6f"
server_key_iv: "b38aef5f14b06ef15e82c3e0d3ca7286"
server_key_version: "encryptionkey@0"
I20260925 22:59:06.133136 10893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:59:06.157970 10893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:59:06.161695 10893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:59:06.161932 10893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:59:06.169234 10893 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.252:45489
I20260925 22:59:06.169541 10961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.252:45489 every 8 connection(s)
I20260925 22:59:06.170276 10893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-2/data/info.pb
I20260925 22:59:06.173272 10962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:59:06.176925 10887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } attempt: 3
I20260925 22:59:06.177345 10962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40539 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:59:06.179073 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 10893
I20260925 22:59:06.179203 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376856092611-20939-0/minicluster-data/master-2/wal/instance
I20260925 22:59:06.179543 20939 external_mini_cluster.cc:1428] Setting key 8a71b700be29b84d901a37f4fe5f9445
Sep 25 22:59:06 dist-test-slave-x1lh krb5kdc[10731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377145, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.253@KRBTEST.COM for kudu/127.20.114.252@KRBTEST.COM
Sep 25 22:59:06 dist-test-slave-x1lh krb5kdc[10731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.252@KRBTEST.COM
Sep 25 22:59:06 dist-test-slave-x1lh krb5kdc[10731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377146, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.252@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:59:06.192242 10890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.177522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:45489 (local address 127.0.0.1:36140)
0925 22:59:06.177717 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:06.177737 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:59:06.177850 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:59:06.180827 (+ 2977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:59:06.180832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:59:06.181092 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:59:06.181211 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:06.181218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.183098 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.183102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:06.183690 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:06.183701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.184519 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.184523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:06.184652 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:06.185249 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:59:06.185268 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:59:06.186766 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:59:06.189855 (+ 3089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:06.189859 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:06.189861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:06.190091 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:06.190479 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:06.190482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:06.190484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:06.190607 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:06.191593 (+ 986us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:59:06.191597 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:59:06.191742 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 22:59:06.191865 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 22:59:06.192046 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 22:59:06.192453 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.179457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37812 (local address 127.20.114.254:40539)
0925 22:59:06.179591 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.179596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.180025 (+ 429us) server_negotiation.cc:407] Connection header received
0925 22:59:06.180751 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.180781 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.180847 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.180934 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.181879 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.182382 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.183618 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.183938 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.184064 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:06.188346 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:06.188351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:06.188355 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:06.188391 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:06.190446 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.190996 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.191001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.191002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.191096 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.191383 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.191386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.191388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.191639 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:06.191753 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:06.192178 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 22:59:06.192310 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:59:06.193787 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.179408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:40539 (local address 127.0.0.1:37812)
0925 22:59:06.179879 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:06.179932 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:59:06.180556 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:59:06.181146 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:59:06.181151 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:59:06.181548 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:59:06.181729 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:06.181739 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.182500 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.182506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:06.183473 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:06.183485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.184111 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.184115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:06.184289 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:06.185545 (+ 1256us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:59:06.185566 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:59:06.188091 (+ 2525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:59:06.190574 (+ 2483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:06.190583 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:06.190590 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:06.190872 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:06.191192 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:06.191197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:06.191199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:06.191286 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:06.191755 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:59:06.191762 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:59:06.191932 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 22:59:06.192597 (+ 665us) client_negotiation.cc:240] Negotiation successful
0925 22:59:06.192842 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":132,"threads_started":1}
I20260925 22:59:06.193794 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.180321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36148 (local address 127.20.114.252:45489)
0925 22:59:06.180636 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.180646 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.180665 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:59:06.180702 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.180705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.180750 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.180910 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.181949 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.182938 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.184024 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.184358 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.184476 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:06.187633 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:06.187637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:06.187640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:06.187669 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:06.190473 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.191063 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.191066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.191068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.191145 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.191481 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.191483 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.191485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.191675 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:06.191866 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:06.192941 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0925 22:59:06.193075 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1089,"server-negotiator.queue_time_us":251,"spinlock_wait_cycles":6400,"thread_start_us":117,"threads_started":1}
I20260925 22:59:06.193794 10964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.178385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36140 (local address 127.20.114.252:45489)
0925 22:59:06.179001 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.179023 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.179071 (+ 48us) server_negotiation.cc:407] Connection header received
0925 22:59:06.180123 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.180151 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.180501 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.180911 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.181432 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.182938 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.183820 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.184367 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.184898 (+ 531us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:06.186995 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:06.187004 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:06.187022 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:06.187050 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:06.189734 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.190216 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.190223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.190230 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.190367 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.190719 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.190723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.190725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.191432 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:06.191609 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:06.192011 (+ 402us) server_negotiation.cc:299] Negotiation successful
0925 22:59:06.192573 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":434,"spinlock_wait_cycles":6144,"thread_start_us":162,"threads_started":1}
I20260925 22:59:06.194200 10812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } attempt: 2
I20260925 22:59:06.196753 10962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:59:06.198983 10887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1051f8735870457e932c20b1c9ea9fc8: Bootstrap starting.
Sep 25 22:59:06 dist-test-slave-x1lh krb5kdc[10731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377145, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for kudu/127.20.114.252@KRBTEST.COM
I20260925 22:59:06.202417 10887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1051f8735870457e932c20b1c9ea9fc8: Neither blocks nor log segments found. Creating new log.
I20260925 22:59:06.203540 10887 log.cc:824] T 00000000000000000000000000000000 P 1051f8735870457e932c20b1c9ea9fc8: Log is configured to *not* fsync() on all Append() calls
I20260925 22:59:06.204234 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.194651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:45489 (local address 127.0.0.1:36156)
0925 22:59:06.194987 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:06.195004 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:59:06.195106 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:59:06.195362 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:59:06.195366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:59:06.195561 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:59:06.195650 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:06.195656 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.196571 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.196574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:06.197364 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:06.197377 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.197985 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.197988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:06.198059 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:06.198588 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:59:06.198603 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:59:06.199932 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:59:06.202354 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:06.202357 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:06.202360 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:06.202696 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:06.203103 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:06.203106 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:06.203108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:06.203210 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:06.203695 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:59:06.203698 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:59:06.203798 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 22:59:06.203885 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 22:59:06.204051 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253}
I20260925 22:59:06.205430 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.194714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36156 (local address 127.20.114.252:45489)
0925 22:59:06.194821 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.194826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.195075 (+ 249us) server_negotiation.cc:407] Connection header received
0925 22:59:06.195190 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.195193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.195232 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.195335 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.195769 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.196433 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.197517 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.197859 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.197967 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:06.200089 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:06.200093 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:06.200096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:06.200123 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:06.202206 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.202891 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.202894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.202896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.202989 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.203328 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.203331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.203333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.203573 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:06.204952 (+ 1379us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:06.205035 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 22:59:06.205170 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 22:59:06.206817 10887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1051f8735870457e932c20b1c9ea9fc8: No bootstrap required, opened a new log
Sep 25 22:59:06 dist-test-slave-x1lh krb5kdc[10731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377146, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.252@KRBTEST.COM for kudu/127.20.114.253@KRBTEST.COM
I20260925 22:59:06.209183 10812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a: Bootstrap starting.
Sep 25 22:59:06 dist-test-slave-x1lh krb5kdc[10731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:59:06.211221 10887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1051f8735870457e932c20b1c9ea9fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927df2298f68408396669d12e2081d5a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40539 } } peers { permanent_uuid: "1051f8735870457e932c20b1c9ea9fc8" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } } peers { permanent_uuid: "8e62d31190a3434587c48dab2d328f7a" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } }
I20260925 22:59:06.211478 10887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1051f8735870457e932c20b1c9ea9fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1051f8735870457e932c20b1c9ea9fc8, State: Initialized, Role: FOLLOWER
I20260925 22:59:06.211818 10887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1051f8735870457e932c20b1c9ea9fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927df2298f68408396669d12e2081d5a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40539 } } peers { permanent_uuid: "1051f8735870457e932c20b1c9ea9fc8" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } } peers { permanent_uuid: "8e62d31190a3434587c48dab2d328f7a" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } }
I20260925 22:59:06.212071 10892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.197469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.20.114.253:36839)
0925 22:59:06.201125 (+ 3656us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.201210 (+ 85us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.201236 (+ 26us) server_negotiation.cc:407] Connection header received
0925 22:59:06.201292 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.201298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.201411 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.201516 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.202109 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.202907 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.204089 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.204469 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.204579 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:06.207891 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:06.207898 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:06.207904 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:06.207942 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:06.209823 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.210396 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.210400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.210402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.210495 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.210886 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.210892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.210895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.211210 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:06.211303 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:06.211736 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 22:59:06.211878 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3538}
I20260925 22:59:06.212248 10812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a: Neither blocks nor log segments found. Creating new log.
I20260925 22:59:06.212699 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.197240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:36839 (local address 127.0.0.1:41042)
0925 22:59:06.197593 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:06.197604 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:59:06.197656 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:59:06.201549 (+ 3893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:59:06.201552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:59:06.201815 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:59:06.201970 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:06.201978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.203049 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.203053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:06.203958 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:06.203970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.204976 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.204980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:06.205094 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:06.205951 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:59:06.205966 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:59:06.207686 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:59:06.209971 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:06.209979 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:06.209984 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:06.210250 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:06.210624 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:06.210628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:06.210632 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:06.210760 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:06.211397 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:59:06.211401 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:59:06.211536 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 22:59:06.212300 (+ 764us) client_negotiation.cc:240] Negotiation successful
0925 22:59:06.212495 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282}
I20260925 22:59:06.213277 10812 log.cc:824] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:59:06.213369 10887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1051f8735870457e932c20b1c9ea9fc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:59:06.213728 10962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:59:06.215035 10969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1051f8735870457e932c20b1c9ea9fc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927df2298f68408396669d12e2081d5a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40539 } } peers { permanent_uuid: "1051f8735870457e932c20b1c9ea9fc8" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } } peers { permanent_uuid: "8e62d31190a3434587c48dab2d328f7a" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } } }
I20260925 22:59:06.215209 10969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1051f8735870457e932c20b1c9ea9fc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:59:06.215798 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.201731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37828 (local address 127.20.114.254:40539)
0925 22:59:06.201872 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.201876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.201891 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:59:06.201932 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.201937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.201977 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.202066 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.203039 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.203814 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.205144 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.205862 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.205955 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:06.210425 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:06.210429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:06.210432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:06.210464 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:06.213127 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.213913 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.213917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.213919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.214014 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.214372 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.214378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.214383 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.214623 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:06.214890 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:06.215491 (+ 601us) server_negotiation.cc:299] Negotiation successful
0925 22:59:06.215624 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 22:59:06.216303 10812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a: No bootstrap required, opened a new log
W20260925 22:59:06.219983 10982 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1051f8735870457e932c20b1c9ea9fc8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:59:06.221153 10982 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:59:06.221340 10812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927df2298f68408396669d12e2081d5a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40539 } } peers { permanent_uuid: "1051f8735870457e932c20b1c9ea9fc8" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } } peers { permanent_uuid: "8e62d31190a3434587c48dab2d328f7a" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } }
I20260925 22:59:06.221571 10812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927df2298f68408396669d12e2081d5a, State: Initialized, Role: FOLLOWER
I20260925 22:59:06.221864 10812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927df2298f68408396669d12e2081d5a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40539 } } peers { permanent_uuid: "1051f8735870457e932c20b1c9ea9fc8" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } } peers { permanent_uuid: "8e62d31190a3434587c48dab2d328f7a" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } }
I20260925 22:59:06.222364 10984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927df2298f68408396669d12e2081d5a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40539 } } peers { permanent_uuid: "1051f8735870457e932c20b1c9ea9fc8" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } } peers { permanent_uuid: "8e62d31190a3434587c48dab2d328f7a" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } } }
I20260925 22:59:06.222525 10984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a [sys.catalog]: This master's current role is: FOLLOWER
Sep 25 22:59:06 dist-test-slave-x1lh krb5kdc[10731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377146, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.252@KRBTEST.COM for kudu/127.20.114.252@KRBTEST.COM
I20260925 22:59:06.223505 10812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 22:59:06 dist-test-slave-x1lh krb5kdc[10731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.253@KRBTEST.COM
I20260925 22:59:06.227226 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.214271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:45489 (local address 127.0.0.1:36164)
0925 22:59:06.214573 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:06.214589 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:59:06.214679 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:59:06.217441 (+ 2762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:59:06.217444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:59:06.217643 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:59:06.217756 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:06.217764 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.218819 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.218822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:06.219676 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:06.219688 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.220347 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.220350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:06.220426 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:06.221510 (+ 1084us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:59:06.221525 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:59:06.223214 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:59:06.225564 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:06.225570 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:06.225575 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:06.225849 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:06.226157 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:06.226160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:06.226162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:06.226239 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:06.226701 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:59:06.226704 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:59:06.226792 (+ 88us) client_negotiation.cc:769] Sending connection context
0925 22:59:06.226892 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 22:59:06.227048 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":96,"threads_started":1}
I20260925 22:59:06.227711 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.214318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36164 (local address 127.20.114.252:45489)
0925 22:59:06.214450 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.214455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.217219 (+ 2764us) server_negotiation.cc:407] Connection header received
0925 22:59:06.217279 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.217281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.217331 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.217433 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.217879 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.218690 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.219826 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.220117 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.220214 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:06.223400 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:06.223403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:06.223406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:06.223433 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:06.225412 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.225965 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.225968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.225970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.226059 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.226340 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.226345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.226349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.226578 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:06.226669 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:06.227448 (+ 779us) server_negotiation.cc:299] Negotiation successful
0925 22:59:06.227565 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:59:06.223104 10819 master_runner.cc:186] Error getting master registration for 127.20.114.254:40539: OK, instance_id { permanent_uuid: "927df2298f68408396669d12e2081d5a" instance_seqno: 1790377145655557 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:59:06.231619 10962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e62d31190a3434587c48dab2d328f7a: Bootstrap starting.
I20260925 22:59:06.234714 10962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e62d31190a3434587c48dab2d328f7a: Neither blocks nor log segments found. Creating new log.
I20260925 22:59:06.235838 10962 log.cc:824] T 00000000000000000000000000000000 P 8e62d31190a3434587c48dab2d328f7a: Log is configured to *not* fsync() on all Append() calls
W20260925 22:59:06.237093 10995 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:59:06.237200 10995 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:59:06.228449 10819 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:59:06.233227 10744 master_runner.cc:186] Error getting master registration for 127.20.114.252:45489: OK, instance_id { permanent_uuid: "8e62d31190a3434587c48dab2d328f7a" instance_seqno: 1790377146167216 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:59:06.238498 10744 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:59:06.239646 10892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.217065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41052 (local address 127.20.114.253:36839)
0925 22:59:06.217175 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.217180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.217719 (+ 539us) server_negotiation.cc:407] Connection header received
0925 22:59:06.217987 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.217990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.218039 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.218091 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.219009 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.219750 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.220845 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.221246 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.221353 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:06.227657 (+ 6304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:06.227662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:06.227665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:06.227697 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:06.236912 (+ 9215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.237820 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.237826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.237828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.237960 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.238403 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.238409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.238412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.238765 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:06.239051 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:06.239326 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 22:59:06.239458 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 22:59:06.241078 10962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e62d31190a3434587c48dab2d328f7a: No bootstrap required, opened a new log
I20260925 22:59:06.246402 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:59:06.246413 10962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e62d31190a3434587c48dab2d328f7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927df2298f68408396669d12e2081d5a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40539 } } peers { permanent_uuid: "1051f8735870457e932c20b1c9ea9fc8" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } } peers { permanent_uuid: "8e62d31190a3434587c48dab2d328f7a" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } }
I20260925 22:59:06.246778 10962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e62d31190a3434587c48dab2d328f7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e62d31190a3434587c48dab2d328f7a, State: Initialized, Role: FOLLOWER
I20260925 22:59:06.247181 10962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e62d31190a3434587c48dab2d328f7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927df2298f68408396669d12e2081d5a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40539 } } peers { permanent_uuid: "1051f8735870457e932c20b1c9ea9fc8" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } } peers { permanent_uuid: "8e62d31190a3434587c48dab2d328f7a" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } }
I20260925 22:59:06.247809 10998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e62d31190a3434587c48dab2d328f7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927df2298f68408396669d12e2081d5a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40539 } } peers { permanent_uuid: "1051f8735870457e932c20b1c9ea9fc8" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } } peers { permanent_uuid: "8e62d31190a3434587c48dab2d328f7a" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } } }
I20260925 22:59:06.247975 10998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e62d31190a3434587c48dab2d328f7a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:59:06.247993 10786 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "927df2298f68408396669d12e2081d5a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37828
I20260925 22:59:06.248271 10786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 22:59:06.248359 10786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:59:06.248811 10962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e62d31190a3434587c48dab2d328f7a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:59:06.249708 10786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927df2298f68408396669d12e2081d5a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40539 } } peers { permanent_uuid: "1051f8735870457e932c20b1c9ea9fc8" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } } peers { permanent_uuid: "8e62d31190a3434587c48dab2d328f7a" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } }
I20260925 22:59:06.251144 10786 leader_election.cc:288] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a [CANDIDATE]: Term 1 election: Requested vote from peers 1051f8735870457e932c20b1c9ea9fc8 (127.20.114.253:36839), 8e62d31190a3434587c48dab2d328f7a (127.20.114.252:45489)
I20260925 22:59:06.252581 10861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "927df2298f68408396669d12e2081d5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1051f8735870457e932c20b1c9ea9fc8"
I20260925 22:59:06.252915 10861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1051f8735870457e932c20b1c9ea9fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:59:06.254585 10861 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1051f8735870457e932c20b1c9ea9fc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927df2298f68408396669d12e2081d5a in term 1.
I20260925 22:59:06.255182 10936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "927df2298f68408396669d12e2081d5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8e62d31190a3434587c48dab2d328f7a"
I20260925 22:59:06.255473 10936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e62d31190a3434587c48dab2d328f7a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:59:06.256999 10936 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e62d31190a3434587c48dab2d328f7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927df2298f68408396669d12e2081d5a in term 1.
I20260925 22:59:06.257860 10762 leader_election.cc:302] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1051f8735870457e932c20b1c9ea9fc8, 927df2298f68408396669d12e2081d5a; no voters:
W20260925 22:59:06.258343 11009 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e62d31190a3434587c48dab2d328f7a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:59:06.258419 11009 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:59:06.260583 10984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:59:06.260912 10984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a [term 1 LEADER]: Becoming Leader. State: Replica: 927df2298f68408396669d12e2081d5a, State: Running, Role: LEADER
I20260925 22:59:06.261363 10984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927df2298f68408396669d12e2081d5a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40539 } } peers { permanent_uuid: "1051f8735870457e932c20b1c9ea9fc8" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } } peers { permanent_uuid: "8e62d31190a3434587c48dab2d328f7a" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } }
I20260925 22:59:06.263288 10984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 927df2298f68408396669d12e2081d5a. Latest consensus state: current_term: 1 leader_uuid: "927df2298f68408396669d12e2081d5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927df2298f68408396669d12e2081d5a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40539 } } peers { permanent_uuid: "1051f8735870457e932c20b1c9ea9fc8" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } } peers { permanent_uuid: "8e62d31190a3434587c48dab2d328f7a" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } } }
I20260925 22:59:06.263446 10984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a [sys.catalog]: This master's current role is: LEADER
I20260925 22:59:06.266677 10893 master_runner.cc:263] No leader master found from master 8e62d31190a3434587c48dab2d328f7a
I20260925 22:59:06.266760 10893 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:59:06.269079 11018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:59:06.270364 11018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:59:06.275368 10892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.260369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41066 (local address 127.20.114.253:36839)
0925 22:59:06.263746 (+ 3377us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.263752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.263773 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:59:06.263828 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.263832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.263893 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.264002 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.266078 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.266827 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.267766 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.268100 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.268207 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:06.270459 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:06.270464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:06.270467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:06.270501 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:06.272832 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.273486 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.273492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.273494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.273600 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.273945 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.273950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.273951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.274212 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:06.274369 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:06.274746 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 22:59:06.275035 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3169}
I20260925 22:59:06.287353 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.268055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36176 (local address 127.20.114.252:45489)
0925 22:59:06.268196 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.268201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.274129 (+ 5928us) server_negotiation.cc:407] Connection header received
0925 22:59:06.274331 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.274335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.274396 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.274455 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.275740 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.276520 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.280023 (+ 3503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.280366 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.280448 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:06.282878 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:06.282883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:06.282886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:06.282922 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:06.285210 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.285830 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.285835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.285837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.285939 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.286263 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.286267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.286269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.286525 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:06.286640 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:06.287036 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 22:59:06.287176 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:59:06.290566 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:06.273316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37838 (local address 127.20.114.254:40539)
0925 22:59:06.273472 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.273476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.276415 (+ 2939us) server_negotiation.cc:407] Connection header received
0925 22:59:06.276613 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.276617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.276680 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.276741 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.277746 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.278741 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.279916 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:06.280244 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:06.280324 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:06.282878 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:06.282883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:06.282886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:06.282922 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:06.285219 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.288583 (+ 3364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.288587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.288589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.288704 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:06.289345 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:06.289349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:06.289351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:06.289629 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:06.289833 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:06.290198 (+ 365us) server_negotiation.cc:299] Negotiation successful
0925 22:59:06.290347 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:59:06.293730 10936 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e62d31190a3434587c48dab2d328f7a [term 1 FOLLOWER]: Refusing update from remote peer 927df2298f68408396669d12e2081d5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:59:06.293848 10861 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1051f8735870457e932c20b1c9ea9fc8 [term 1 FOLLOWER]: Refusing update from remote peer 927df2298f68408396669d12e2081d5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:59:06.294803 11016 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1051f8735870457e932c20b1c9ea9fc8" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:59:06.295171 10984 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e62d31190a3434587c48dab2d328f7a" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:59:06.301426 10998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e62d31190a3434587c48dab2d328f7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 927df2298f68408396669d12e2081d5a. Latest consensus state: current_term: 1 leader_uuid: "927df2298f68408396669d12e2081d5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927df2298f68408396669d12e2081d5a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40539 } } peers { permanent_uuid: "1051f8735870457e932c20b1c9ea9fc8" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } } peers { permanent_uuid: "8e62d31190a3434587c48dab2d328f7a" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } } }
I20260925 22:59:06.301496 10969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1051f8735870457e932c20b1c9ea9fc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 927df2298f68408396669d12e2081d5a. Latest consensus state: current_term: 1 leader_uuid: "927df2298f68408396669d12e2081d5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927df2298f68408396669d12e2081d5a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40539 } } peers { permanent_uuid: "1051f8735870457e932c20b1c9ea9fc8" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } } peers { permanent_uuid: "8e62d31190a3434587c48dab2d328f7a" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } } }
I20260925 22:59:06.301615 10998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e62d31190a3434587c48dab2d328f7a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:59:06.301626 10969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1051f8735870457e932c20b1c9ea9fc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:59:06.307060 10984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "927df2298f68408396669d12e2081d5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927df2298f68408396669d12e2081d5a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40539 } } peers { permanent_uuid: "1051f8735870457e932c20b1c9ea9fc8" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } } peers { permanent_uuid: "8e62d31190a3434587c48dab2d328f7a" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } } }
I20260925 22:59:06.307245 10984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a [sys.catalog]: This master's current role is: LEADER
I20260925 22:59:06.309134 10969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1051f8735870457e932c20b1c9ea9fc8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "927df2298f68408396669d12e2081d5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927df2298f68408396669d12e2081d5a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40539 } } peers { permanent_uuid: "1051f8735870457e932c20b1c9ea9fc8" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } } peers { permanent_uuid: "8e62d31190a3434587c48dab2d328f7a" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } } }
I20260925 22:59:06.309306 10969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1051f8735870457e932c20b1c9ea9fc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:59:06.311350 11021 mvcc.cc:205] Tried to move back new op lower bound from 7333384791192559616 to 7333384791090524160. Current Snapshot: MvccSnapshot[applied={T|T < 7333384791192559616}]
I20260925 22:59:06.312213 11023 mvcc.cc:205] Tried to move back new op lower bound from 7333384791192559616 to 7333384791090524160. Current Snapshot: MvccSnapshot[applied={T|T < 7333384791192559616}]
I20260925 22:59:06.313665 11018 catalog_manager.cc:1347] Generated new cluster ID: 9123dab228ed464e8bc02b47bb2222ac
I20260925 22:59:06.313748 11018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:59:06.317360 11016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "927df2298f68408396669d12e2081d5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927df2298f68408396669d12e2081d5a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40539 } } peers { permanent_uuid: "1051f8735870457e932c20b1c9ea9fc8" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } } peers { permanent_uuid: "8e62d31190a3434587c48dab2d328f7a" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } } }
I20260925 22:59:06.317543 11016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a [sys.catalog]: This master's current role is: LEADER
I20260925 22:59:06.320930 10998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e62d31190a3434587c48dab2d328f7a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "927df2298f68408396669d12e2081d5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927df2298f68408396669d12e2081d5a" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 40539 } } peers { permanent_uuid: "1051f8735870457e932c20b1c9ea9fc8" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 36839 } } peers { permanent_uuid: "8e62d31190a3434587c48dab2d328f7a" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 45489 } } }
I20260925 22:59:06.321086 10998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e62d31190a3434587c48dab2d328f7a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:59:06.322384 11022 mvcc.cc:205] Tried to move back new op lower bound from 7333384791192559616 to 7333384791090524160. Current Snapshot: MvccSnapshot[applied={T|T < 7333384791192559616}]
I20260925 22:59:06.331449 11018 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:59:06.332854 11018 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:59:06.342211 11018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 927df2298f68408396669d12e2081d5a: Generated new TSK 0
I20260925 22:59:06.343374 11018 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:59:06.385150 10816 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:06.381026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37850 (local address 127.20.114.254:40539)
0925 22:59:06.381180 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.381185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.383800 (+ 2615us) server_negotiation.cc:407] Connection header received
0925 22:59:06.384030 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.384034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.384095 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.384152 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.384993 (+ 841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37850: BlockingRecv error: recv got EOF from 127.0.0.1:37850 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 22:59:06.386933 10967 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:06.382104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36192 (local address 127.20.114.252:45489)
0925 22:59:06.382244 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.382249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.385917 (+ 3668us) server_negotiation.cc:407] Connection header received
0925 22:59:06.386109 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.386112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.386165 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.386216 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.386790 (+ 574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36192: BlockingRecv error: recv got EOF from 127.0.0.1:36192 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 22:59:06.388790 10892 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:06.381615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41068 (local address 127.20.114.253:36839)
0925 22:59:06.381742 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.381747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.387645 (+ 5898us) server_negotiation.cc:407] Connection header received
0925 22:59:06.387832 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.387836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.387893 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.387946 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.388622 (+ 676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41068: BlockingRecv error: recv got EOF from 127.0.0.1:41068 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 22:59:06.394708 10892 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:06.391731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41076 (local address 127.20.114.253:36839)
0925 22:59:06.391863 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.391868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.393207 (+ 1339us) server_negotiation.cc:407] Connection header received
0925 22:59:06.393497 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.393501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.393592 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.393649 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.394579 (+ 930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41076: BlockingRecv error: recv got EOF from 127.0.0.1:41076 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 22:59:06.400616 10892 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:06.398759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41078 (local address 127.20.114.253:36839)
0925 22:59:06.398890 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.398895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.399552 (+ 657us) server_negotiation.cc:407] Connection header received
0925 22:59:06.399764 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.399768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.399825 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.399879 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.400490 (+ 611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41078: BlockingRecv error: recv got EOF from 127.0.0.1:41078 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 22:59:06.407297 10892 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:06.405518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.20.114.253:36839)
0925 22:59:06.405641 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.405646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.406257 (+ 611us) server_negotiation.cc:407] Connection header received
0925 22:59:06.406475 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.406479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.406532 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.406586 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.407160 (+ 574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41080: BlockingRecv error: recv got EOF from 127.0.0.1:41080 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 22:59:06.415778 10892 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:06.413971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41082 (local address 127.20.114.253:36839)
0925 22:59:06.414121 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.414125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.414724 (+ 599us) server_negotiation.cc:407] Connection header received
0925 22:59:06.414933 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.414937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.414990 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.415043 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.415656 (+ 613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41082: BlockingRecv error: recv got EOF from 127.0.0.1:41082 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260925 22:59:06.428701 10892 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:06.426774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41086 (local address 127.20.114.253:36839)
0925 22:59:06.426905 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.426909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.427588 (+ 679us) server_negotiation.cc:407] Connection header received
0925 22:59:06.427814 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.427818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.427876 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.427947 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.428578 (+ 631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41086: BlockingRecv error: recv got EOF from 127.0.0.1:41086 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 22:59:06.450374 10892 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:06.447722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.20.114.253:36839)
0925 22:59:06.447864 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.447869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.448815 (+ 946us) server_negotiation.cc:407] Connection header received
0925 22:59:06.449129 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.449134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.449228 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.449302 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.450196 (+ 894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41092: BlockingRecv error: recv got EOF from 127.0.0.1:41092 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 22:59:06.487375 10892 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:06.485327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41094 (local address 127.20.114.253:36839)
0925 22:59:06.485471 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.485476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.486152 (+ 676us) server_negotiation.cc:407] Connection header received
0925 22:59:06.486451 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.486455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.486531 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.486590 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.487248 (+ 658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41094: BlockingRecv error: recv got EOF from 127.0.0.1:41094 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 22:59:06.555456 10892 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:06.554164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41110 (local address 127.20.114.253:36839)
0925 22:59:06.554346 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.554352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.554528 (+ 176us) server_negotiation.cc:407] Connection header received
0925 22:59:06.554626 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.554628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.554682 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.554767 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.555327 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41110: BlockingRecv error: recv got EOF from 127.0.0.1:41110 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260925 22:59:06.687741 10892 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:06.686425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41120 (local address 127.20.114.253:36839)
0925 22:59:06.686566 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.686570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.686707 (+ 137us) server_negotiation.cc:407] Connection header received
0925 22:59:06.686836 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.686839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.686935 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.687020 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.687603 (+ 583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41120: BlockingRecv error: recv got EOF from 127.0.0.1:41120 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 22:59:06.947537 10892 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:06.946353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41126 (local address 127.20.114.253:36839)
0925 22:59:06.946512 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:06.946516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:06.946574 (+ 58us) server_negotiation.cc:407] Connection header received
0925 22:59:06.946738 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:06.946741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:06.946792 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:06.946937 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:06.947434 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41126: BlockingRecv error: recv got EOF from 127.0.0.1:41126 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 22:59:07.222534 10982 catalog_manager.cc:1259] Loaded cluster ID: 9123dab228ed464e8bc02b47bb2222ac
I20260925 22:59:07.222628 10982 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1051f8735870457e932c20b1c9ea9fc8: loading cluster ID for follower catalog manager: success
I20260925 22:59:07.224581 10982 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1051f8735870457e932c20b1c9ea9fc8: acquiring CA information for follower catalog manager: success
I20260925 22:59:07.225970 10982 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1051f8735870457e932c20b1c9ea9fc8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:59:07.259763 11009 catalog_manager.cc:1259] Loaded cluster ID: 9123dab228ed464e8bc02b47bb2222ac
I20260925 22:59:07.259876 11009 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8e62d31190a3434587c48dab2d328f7a: loading cluster ID for follower catalog manager: success
I20260925 22:59:07.261680 11009 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8e62d31190a3434587c48dab2d328f7a: acquiring CA information for follower catalog manager: success
I20260925 22:59:07.263465 11009 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8e62d31190a3434587c48dab2d328f7a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:59:07.466229 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:07.462001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41128 (local address 127.20.114.253:36839)
0925 22:59:07.462291 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:07.462296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:07.462313 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:59:07.462530 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:07.462532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:07.462665 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:07.462787 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:59:07.463085 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:07.463628 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:07.464539 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:07.464779 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:07.464924 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:07.465286 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:59:07.465765 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:59:07.465918 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:07.465984 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 22:59:07.466053 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":81,"threads_started":1}
I20260925 22:59:07.474078 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:07.469765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.20.114.252:45489)
0925 22:59:07.470036 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:07.470040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:07.470056 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:59:07.470146 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:07.470148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:07.470270 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:07.470400 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:59:07.470676 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:07.471215 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:07.472285 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:07.472516 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:07.472599 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:07.473030 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:59:07.473505 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:59:07.473616 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:07.473718 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 22:59:07.473829 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":77,"threads_started":1}
I20260925 22:59:07.482656 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:07.478146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41142 (local address 127.20.114.253:36839)
0925 22:59:07.478349 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:07.478353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:07.478370 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:59:07.478434 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:07.478436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:07.478482 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:07.478617 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:59:07.478901 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:07.479676 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:07.481062 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:07.481427 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:07.481537 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:07.481917 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:59:07.482019 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:59:07.482091 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:07.482447 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 22:59:07.482512 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:59:07.483330 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:07.478011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36214 (local address 127.20.114.252:45489)
0925 22:59:07.478171 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:07.478175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:07.479254 (+ 1079us) server_negotiation.cc:407] Connection header received
0925 22:59:07.479351 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:07.479354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:07.479407 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:07.479483 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:59:07.479747 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:07.480293 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:07.481739 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:07.482094 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:07.482213 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:07.482536 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:59:07.482626 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:59:07.482664 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:07.483125 (+ 461us) server_negotiation.cc:299] Negotiation successful
0925 22:59:07.483190 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 22:59:07.483341 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:07.477794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37860 (local address 127.20.114.254:40539)
0925 22:59:07.478606 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:07.478609 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:07.478624 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:59:07.478711 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:07.478713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:07.478909 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:07.479042 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:59:07.479438 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:07.480306 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:07.481530 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:07.481866 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:07.481977 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:07.482339 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:59:07.482745 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:59:07.482841 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:07.483009 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 22:59:07.483075 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":93,"threads_started":1}
I20260925 22:59:07.487522 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 10744
I20260925 22:59:07.494722 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 10819
I20260925 22:59:07.502849 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 10893
2026-09-25T22:59:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3267 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:59:07 dist-test-slave-x1lh krb5kdc[11113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:59:07 dist-test-slave-x1lh krb5kdc[11113](info): set up 2 sockets
Sep 25 22:59:07 dist-test-slave-x1lh krb5kdc[11113](info): commencing operation
krb5kdc: starting...
W20260925 22:59:08.573689 20939 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/test-user.keytab.
Sep 25 22:59:08 dist-test-slave-x1lh krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:59:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:59:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.114.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
Entry for principal kudu/127.20.114.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab.
I20260925 22:59:08.720233 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:35573
--webserver_interface=127.20.114.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:43803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:35573,127.20.114.253:39815,127.20.114.252:40271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:59:08.845327 11126 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:59:08.845664 11126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:59:08.845688 11126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:59:08.845785 11126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:59:08.850178 11126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:59:08.850262 11126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:59:08.850283 11126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:59:08.850301 11126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:59:08.850317 11126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:59:08.856541 11126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:43803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:35573,127.20.114.253:39815,127.20.114.252:40271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.254.keytab
--principal=kudu/127.20.114.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.254:35573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.114.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:59:08.857872 11126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:59:08.859036 11126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:59:08.868443 11131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:59:08.868629 11132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:59:08.868453 11134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:59:08.868804 11126 server_base.cc:1034] running on GCE node
I20260925 22:59:08.869557 11126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:59:08.870949 11126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:59:08.872161 11126 hybrid_clock.cc:648] HybridClock initialized: now 1790377148872149 us; error 37 us; skew 500 ppm
Sep 25 22:59:08 dist-test-slave-x1lh krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377148, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:59:08.875218 11126 init.cc:377] Logged in from keytab as kudu/127.20.114.254@KRBTEST.COM (short username kudu)
I20260925 22:59:08.876319 11126 webserver.cc:460] Webserver started at http://127.20.114.254:41069/ using document root <none> and password file <none>
I20260925 22:59:08.877028 11126 fs_manager.cc:359] Metadata directory not provided
I20260925 22:59:08.877132 11126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:59:08.877395 11126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:59:08.879635 11126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-0/data/instance:
uuid: "5229d4e7383442cd8ad8740d802c5014"
format_stamp: "Formatted at 2026-09-25 22:59:08 on dist-test-slave-x1lh"
server_key: "ba4bce01e7976149e50ec2381d796b6c"
server_key_iv: "cb8293dc3e6494de118c7cbaf2d33d12"
server_key_version: "encryptionkey@0"
I20260925 22:59:08.880271 11126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-0/wal/instance:
uuid: "5229d4e7383442cd8ad8740d802c5014"
format_stamp: "Formatted at 2026-09-25 22:59:08 on dist-test-slave-x1lh"
server_key: "ba4bce01e7976149e50ec2381d796b6c"
server_key_iv: "cb8293dc3e6494de118c7cbaf2d33d12"
server_key_version: "encryptionkey@0"
I20260925 22:59:08.884663 11126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:59:08.888077 11141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:59:08.889827 11126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:59:08.890023 11126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-0/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-0/wal
uuid: "5229d4e7383442cd8ad8740d802c5014"
format_stamp: "Formatted at 2026-09-25 22:59:08 on dist-test-slave-x1lh"
server_key: "ba4bce01e7976149e50ec2381d796b6c"
server_key_iv: "cb8293dc3e6494de118c7cbaf2d33d12"
server_key_version: "encryptionkey@0"
I20260925 22:59:08.890177 11126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:59:08.924383 11126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:59:08.927945 11126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:59:08.928205 11126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:59:08.935649 11126 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.254:35573
I20260925 22:59:08.935989 11193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.254:35573 every 8 connection(s)
I20260925 22:59:08.936753 11126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-0/data/info.pb
I20260925 22:59:08.939065 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 11126
I20260925 22:59:08.939222 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-0/wal/instance
I20260925 22:59:08.939545 20939 external_mini_cluster.cc:1428] Setting key 9061e42bcdbd4b63cf24e81237534146
I20260925 22:59:08.940043 11194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:59:08.944274 11194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35573 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.114.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.253@KRBTEST.COM" created.
Sep 25 22:59:08 dist-test-slave-x1lh krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377148, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:59:08.962281 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:08.946883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39594 (local address 127.20.114.254:35573)
0925 22:59:08.947502 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:08.947519 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:08.947705 (+ 186us) server_negotiation.cc:407] Connection header received
0925 22:59:08.949139 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:08.949145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:08.949384 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:08.949561 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:08.950726 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:08.951881 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:08.953343 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:08.953754 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:08.953942 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:08.956785 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:08.956793 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:08.956811 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:08.956836 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:08.958792 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:08.959439 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:08.959445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:08.959446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:08.959561 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:08.959918 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:08.959922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:08.959923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:08.960521 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:08.960645 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:08.961258 (+ 613us) server_negotiation.cc:299] Negotiation successful
0925 22:59:08.961464 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":185,"threads_started":1}
I20260925 22:59:08.962281 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:08.946883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:35573 (local address 127.0.0.1:39594)
0925 22:59:08.947440 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:08.947499 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:59:08.948718 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:59:08.949629 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:59:08.949634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:59:08.950176 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:59:08.950453 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:08.950468 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:08.952033 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:08.952039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:08.953197 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:08.953207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:08.953922 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:08.953925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:08.954021 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:08.954687 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:59:08.954705 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:59:08.956586 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:59:08.958924 (+ 2338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:08.958935 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:08.958947 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:08.959296 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:08.959663 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:08.959666 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:08.959667 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:08.959792 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:08.960653 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:59:08.960658 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:59:08.960791 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 22:59:08.961042 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 22:59:08.961365 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":149,"threads_started":1}
I20260925 22:59:08.965251 11194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:59:08.966223 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:08.965869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:39815 (local address 127.0.0.1:46354)
0925 22:59:08.966016 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:08.966142 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.253:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 22:59:08.966501 11143 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.114.253:39815: connect: Connection refused (error 111)
I20260925 22:59:08.967561 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:08.967267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:39815 (local address 127.0.0.1:46358)
0925 22:59:08.967434 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:08.967499 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.253:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 22:59:08.967712 11194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.253:39815: Network error: Client connection negotiation failed: client connection to 127.20.114.253:39815: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.253.keytab.
Entry for principal kudu/127.20.114.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.253.keytab.
I20260925 22:59:08.973917 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.253:39815
--webserver_interface=127.20.114.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.253.keytab
--principal=kudu/127.20.114.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:43803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:35573,127.20.114.253:39815,127.20.114.252:40271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:59:09.017012 11194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } attempt: 1
I20260925 22:59:09.018004 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.017648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:39815 (local address 127.0.0.1:46362)
0925 22:59:09.017820 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.017922 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.253:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 22:59:09.018779 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.018553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:39815 (local address 127.0.0.1:46364)
0925 22:59:09.018663 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.018718 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.253:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 22:59:09.018924 11194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.253:39815: Network error: Client connection negotiation failed: client connection to 127.20.114.253:39815: connect: Connection refused (error 111)
I20260925 22:59:09.087133 11194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } attempt: 2
I20260925 22:59:09.088127 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.087757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:39815 (local address 127.0.0.1:46374)
0925 22:59:09.087979 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.088046 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.253:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260925 22:59:09.088973 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.088678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:39815 (local address 127.0.0.1:46390)
0925 22:59:09.088791 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.088848 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.253:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 22:59:09.089139 11194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.253:39815: Network error: Client connection negotiation failed: client connection to 127.20.114.253:39815: connect: Connection refused (error 111)
W20260925 22:59:09.094712 11201 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:59:09.095114 11201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:59:09.095181 11201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:59:09.095261 11201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:59:09.099962 11201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:59:09.100068 11201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:59:09.100091 11201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:59:09.100109 11201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:59:09.100126 11201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:59:09.105926 11201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:43803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:35573,127.20.114.253:39815,127.20.114.252:40271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.253.keytab
--principal=kudu/127.20.114.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.253:39815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.114.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:59:09.107030 11201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:59:09.108047 11201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:59:09.117594 11206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:59:09.117596 11209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:59:09.117596 11207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:59:09.118175 11201 server_base.cc:1034] running on GCE node
I20260925 22:59:09.118839 11201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:59:09.120082 11201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:59:09.121260 11201 hybrid_clock.cc:648] HybridClock initialized: now 1790377149121223 us; error 62 us; skew 500 ppm
Sep 25 22:59:09 dist-test-slave-x1lh krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377149, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:59:09.124679 11201 init.cc:377] Logged in from keytab as kudu/127.20.114.253@KRBTEST.COM (short username kudu)
I20260925 22:59:09.125824 11201 webserver.cc:460] Webserver started at http://127.20.114.253:38521/ using document root <none> and password file <none>
I20260925 22:59:09.126452 11201 fs_manager.cc:359] Metadata directory not provided
I20260925 22:59:09.126523 11201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:59:09.126756 11201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:59:09.129417 11201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-1/data/instance:
uuid: "50a842aa7ad84143951fc569a5823078"
format_stamp: "Formatted at 2026-09-25 22:59:09 on dist-test-slave-x1lh"
server_key: "b92e089534ed95621394b1ab1752ff6e"
server_key_iv: "e8a364393648107278fecebb31049daf"
server_key_version: "encryptionkey@0"
I20260925 22:59:09.130110 11201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-1/wal/instance:
uuid: "50a842aa7ad84143951fc569a5823078"
format_stamp: "Formatted at 2026-09-25 22:59:09 on dist-test-slave-x1lh"
server_key: "b92e089534ed95621394b1ab1752ff6e"
server_key_iv: "e8a364393648107278fecebb31049daf"
server_key_version: "encryptionkey@0"
I20260925 22:59:09.134461 11201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:59:09.137552 11216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:59:09.139264 11201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:59:09.139472 11201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-1/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-1/wal
uuid: "50a842aa7ad84143951fc569a5823078"
format_stamp: "Formatted at 2026-09-25 22:59:09 on dist-test-slave-x1lh"
server_key: "b92e089534ed95621394b1ab1752ff6e"
server_key_iv: "e8a364393648107278fecebb31049daf"
server_key_version: "encryptionkey@0"
I20260925 22:59:09.139621 11201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:59:09.169587 11201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:59:09.173342 11201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:59:09.173595 11201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:59:09.180320 11194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } attempt: 3
I20260925 22:59:09.181306 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.180949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:39815 (local address 127.0.0.1:46394)
0925 22:59:09.181121 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.181198 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.253:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 22:59:09.181555 11201 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.253:39815
I20260925 22:59:09.181897 11268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.253:39815 every 8 connection(s)
I20260925 22:59:09.182870 11201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-1/data/info.pb
I20260925 22:59:09.186347 11270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 22:59:09 dist-test-slave-x1lh krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377148, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for kudu/127.20.114.253@KRBTEST.COM
I20260925 22:59:09.191723 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 11201
I20260925 22:59:09.191380 11270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35573 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:59:09.191879 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-1/wal/instance
I20260925 22:59:09.192157 20939 external_mini_cluster.cc:1428] Setting key 930422bf1ec7bf4839be9b813d78d544
I20260925 22:59:09.196722 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.181937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:39815 (local address 127.0.0.1:46408)
0925 22:59:09.182064 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.182079 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:59:09.182180 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:59:09.185353 (+ 3173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:59:09.185356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:59:09.185531 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:59:09.185656 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:09.185662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.187547 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.187551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:09.188154 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:09.188168 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.189239 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.189243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:09.189344 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:09.189958 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:59:09.189973 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:59:09.191573 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:59:09.194744 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:09.194749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:09.194751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:09.194990 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:09.195404 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:09.195407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:09.195409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:09.195508 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:09.196159 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:59:09.196162 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:59:09.196281 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 22:59:09.196394 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 22:59:09.196547 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 22:59:09.197942 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.182929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.20.114.253:39815)
0925 22:59:09.183444 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.183464 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.183510 (+ 46us) server_negotiation.cc:407] Connection header received
0925 22:59:09.184445 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.184468 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.184980 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.185396 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.185825 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.187352 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.188313 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.189070 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.189237 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:09.191929 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:09.191939 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:09.191960 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:09.192013 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:09.194630 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.195139 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.195145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.195152 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.195291 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.195606 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.195611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.195613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.196037 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:09.196180 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:09.196548 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 22:59:09.196940 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":133,"threads_started":1}
I20260925 22:59:09.200623 11194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:59:09 dist-test-slave-x1lh krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377149, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.253@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:59:09.201628 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.201261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:40271 (local address 127.0.0.1:33734)
0925 22:59:09.201467 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.201537 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
WARNING: no policy specified for kudu/127.20.114.252@KRBTEST.COM; defaulting to no policy
I20260925 22:59:09.203055 11274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.202138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:40271 (local address 127.0.0.1:33744)
0925 22:59:09.202910 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.202960 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":192,"threads_started":1}
W20260925 22:59:09.203217 11194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.252:40271: Network error: Client connection negotiation failed: client connection to 127.20.114.252:40271: connect: Connection refused (error 111)
I20260925 22:59:09.205965 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.193889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39596 (local address 127.20.114.254:35573)
0925 22:59:09.194050 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.194055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.194485 (+ 430us) server_negotiation.cc:407] Connection header received
0925 22:59:09.194716 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.194720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.194790 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.194850 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.195783 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.196636 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.198055 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.198852 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.198983 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:09.201826 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:09.201832 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:09.201836 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:09.201864 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:09.203666 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.204333 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.204339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.204346 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.204427 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.204926 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.204930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.204934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.205166 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:09.205278 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:09.205694 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 22:59:09.205818 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 22:59:09.205999 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.193869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:35573 (local address 127.0.0.1:39596)
0925 22:59:09.194334 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.194400 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 22:59:09.194597 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:59:09.195053 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:59:09.195060 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:59:09.195507 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:59:09.195640 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:09.195646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.196786 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.196793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:09.197915 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:09.197930 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.198985 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.198989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:09.199134 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:09.199830 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:59:09.199845 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:59:09.201638 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:59:09.203838 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:09.203850 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:09.203857 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:09.204192 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:09.204559 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:09.204561 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:09.204563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:09.204654 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:09.205318 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:59:09.205325 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:59:09.205458 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 22:59:09.205598 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 22:59:09.205842 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.114.252@KRBTEST.COM" created.
I20260925 22:59:09.207442 11270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:59:09 dist-test-slave-x1lh krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377149, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.253@KRBTEST.COM for kudu/127.20.114.253@KRBTEST.COM
I20260925 22:59:09.217262 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.208155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:39815 (local address 127.0.0.1:46418)
0925 22:59:09.208419 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.208430 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:59:09.208562 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:59:09.208857 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:59:09.208894 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:59:09.209102 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:59:09.209208 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:09.209214 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.210065 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.210072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:09.210669 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:09.210676 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.211189 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.211192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:09.211293 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:09.211964 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:59:09.211977 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:59:09.213447 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:59:09.215515 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:09.215521 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:09.215523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:09.215742 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:09.216080 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:09.216083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:09.216086 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:09.216173 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:09.216617 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:59:09.216621 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:59:09.216724 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 22:59:09.216847 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 22:59:09.217053 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225}
I20260925 22:59:09.217262 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.208198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46418 (local address 127.20.114.253:39815)
0925 22:59:09.208368 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.208372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.208505 (+ 133us) server_negotiation.cc:407] Connection header received
0925 22:59:09.208680 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.208684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.208726 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.208824 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.209329 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.209945 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.210793 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.211063 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.211186 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:09.213622 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:09.213625 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:09.213627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:09.213649 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:09.215358 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.215880 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.215888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.215890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.215963 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.216279 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.216284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.216286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.216493 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:09.216633 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:09.216952 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 22:59:09.217117 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":96}
I20260925 22:59:09.218518 11270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:59:09.219610 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.219197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:40271 (local address 127.0.0.1:33748)
0925 22:59:09.219358 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.219509 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260925 22:59:09.220011 11219 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.114.252:40271: connect: Connection refused (error 111)
I20260925 22:59:09.221227 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.220854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:40271 (local address 127.0.0.1:33762)
0925 22:59:09.221080 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.221152 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260925 22:59:09.221467 11270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.252:40271: Network error: Client connection negotiation failed: client connection to 127.20.114.252:40271: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.114.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.252.keytab.
Entry for principal kudu/127.20.114.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.252.keytab.
I20260925 22:59:09.226052 20939 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
/tmp/dist-test-taskV8JYeA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.252:40271
--webserver_interface=127.20.114.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.252.keytab
--principal=kudu/127.20.114.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.114.212:43803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.114.254:35573,127.20.114.253:39815,127.20.114.252:40271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:59:09.234426 11194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } attempt: 1
I20260925 22:59:09.235670 11274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.235148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:40271 (local address 127.0.0.1:33764)
0925 22:59:09.235451 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.235544 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260925 22:59:09.236675 11274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.236360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:40271 (local address 127.0.0.1:33778)
0925 22:59:09.236508 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.236578 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 22:59:09.236893 11194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.252:40271: Network error: Client connection negotiation failed: client connection to 127.20.114.252:40271: connect: Connection refused (error 111)
I20260925 22:59:09.270762 11270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } attempt: 1
I20260925 22:59:09.271751 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.271445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:40271 (local address 127.0.0.1:33786)
0925 22:59:09.271596 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.271668 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 22:59:09.272569 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.272327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:40271 (local address 127.0.0.1:33800)
0925 22:59:09.272453 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.272510 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 22:59:09.272713 11270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.252:40271: Network error: Client connection negotiation failed: client connection to 127.20.114.252:40271: connect: Connection refused (error 111)
I20260925 22:59:09.312109 11194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } attempt: 2
I20260925 22:59:09.313397 11274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.312995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:40271 (local address 127.0.0.1:33802)
0925 22:59:09.313214 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.313293 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260925 22:59:09.314371 11274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.314128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:40271 (local address 127.0.0.1:33806)
0925 22:59:09.314259 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.314310 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 22:59:09.314563 11194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.252:40271: Network error: Client connection negotiation failed: client connection to 127.20.114.252:40271: connect: Connection refused (error 111)
I20260925 22:59:09.340967 11270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } attempt: 2
I20260925 22:59:09.341936 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.341595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:40271 (local address 127.0.0.1:33808)
0925 22:59:09.341774 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.341851 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 22:59:09.342908 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.342597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:40271 (local address 127.0.0.1:33824)
0925 22:59:09.342742 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.342830 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260925 22:59:09.343056 11270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.252:40271: Network error: Client connection negotiation failed: client connection to 127.20.114.252:40271: connect: Connection refused (error 111)
W20260925 22:59:09.352068 11277 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:59:09.352481 11277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:59:09.352560 11277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:59:09.352648 11277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:59:09.357012 11277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:59:09.357128 11277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:59:09.357162 11277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:59:09.357203 11277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:59:09.357239 11277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:59:09.365566 11277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.114.212:43803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.114.254:35573,127.20.114.253:39815,127.20.114.252:40271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/krb5kdc/kudu_127.20.114.252.keytab
--principal=kudu/127.20.114.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.114.252:40271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.114.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:59:09.367259 11277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:59:09.368700 11277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:59:09.378243 11283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:59:09.378533 11277 server_base.cc:1034] running on GCE node
W20260925 22:59:09.378291 11285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:59:09.378243 11282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:59:09.379367 11277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:59:09.380745 11277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:59:09.381975 11277 hybrid_clock.cc:648] HybridClock initialized: now 1790377149381952 us; error 40 us; skew 500 ppm
Sep 25 22:59:09 dist-test-slave-x1lh krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377149, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:59:09.385732 11277 init.cc:377] Logged in from keytab as kudu/127.20.114.252@KRBTEST.COM (short username kudu)
I20260925 22:59:09.386787 11277 webserver.cc:460] Webserver started at http://127.20.114.252:42951/ using document root <none> and password file <none>
I20260925 22:59:09.387432 11277 fs_manager.cc:359] Metadata directory not provided
I20260925 22:59:09.387502 11277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:59:09.387743 11277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:59:09.390097 11277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-2/data/instance:
uuid: "4bfa3908a18640e0bc470ed87083afea"
format_stamp: "Formatted at 2026-09-25 22:59:09 on dist-test-slave-x1lh"
server_key: "5b48a34f1dea06ebb0a9c2ccc080fa15"
server_key_iv: "f7d9ca1b34d1cb73beec2f88f289e36f"
server_key_version: "encryptionkey@0"
I20260925 22:59:09.390729 11277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-2/wal/instance:
uuid: "4bfa3908a18640e0bc470ed87083afea"
format_stamp: "Formatted at 2026-09-25 22:59:09 on dist-test-slave-x1lh"
server_key: "5b48a34f1dea06ebb0a9c2ccc080fa15"
server_key_iv: "f7d9ca1b34d1cb73beec2f88f289e36f"
server_key_version: "encryptionkey@0"
I20260925 22:59:09.395392 11277 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:59:09.398301 11292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:59:09.399823 11277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:59:09.400004 11277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-2/data,/tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-2/wal
uuid: "4bfa3908a18640e0bc470ed87083afea"
format_stamp: "Formatted at 2026-09-25 22:59:09 on dist-test-slave-x1lh"
server_key: "5b48a34f1dea06ebb0a9c2ccc080fa15"
server_key_iv: "f7d9ca1b34d1cb73beec2f88f289e36f"
server_key_version: "encryptionkey@0"
I20260925 22:59:09.400149 11277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:59:09.413847 11194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } attempt: 3
I20260925 22:59:09.414717 11274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.414400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:40271 (local address 127.0.0.1:33836)
0925 22:59:09.414565 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.414628 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 22:59:09.415740 11274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.415409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:40271 (local address 127.0.0.1:33840)
0925 22:59:09.415607 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.415668 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.114.252:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260925 22:59:09.416006 11194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.114.252:40271: Network error: Client connection negotiation failed: client connection to 127.20.114.252:40271: connect: Connection refused (error 111)
I20260925 22:59:09.416968 11277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:59:09.420702 11277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:59:09.420974 11277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:59:09.428709 11277 rpc_server.cc:307] RPC server started. Bound to: 127.20.114.252:40271
I20260925 22:59:09.429066 11344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.114.252:40271 every 8 connection(s)
I20260925 22:59:09.429867 11277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-2/data/info.pb
I20260925 22:59:09.432792 11345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:59:09.434299 20939 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu as pid 11277
I20260925 22:59:09.434247 11270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } attempt: 3
I20260925 22:59:09.434435 20939 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV8JYeA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376856092611-20939-0/minicluster-data/master-2/wal/instance
I20260925 22:59:09.434924 20939 external_mini_cluster.cc:1428] Setting key 7162896537c02cc19a83e8e6eaaad03f
I20260925 22:59:09.438110 11345 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35573 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:59:09 dist-test-slave-x1lh krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377149, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.253@KRBTEST.COM for kudu/127.20.114.252@KRBTEST.COM
Sep 25 22:59:09 dist-test-slave-x1lh krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.252@KRBTEST.COM
Sep 25 22:59:09 dist-test-slave-x1lh krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377149, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.252@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:59:09.450606 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.434895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:40271 (local address 127.0.0.1:33852)
0925 22:59:09.435144 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.435163 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:59:09.435267 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:59:09.438125 (+ 2858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:59:09.438130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:59:09.438402 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:59:09.438555 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:09.438565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.440536 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.440540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:09.441540 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:09.441555 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.442541 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.442545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:09.442636 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:09.443243 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:59:09.443267 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:59:09.445168 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:59:09.448390 (+ 3222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:09.448398 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:09.448403 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:09.448792 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:09.449178 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:09.449181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:09.449183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:09.449297 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:09.450014 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:59:09.450020 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:59:09.450142 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 22:59:09.450236 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 22:59:09.450416 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260925 22:59:09.452001 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.440296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39602 (local address 127.20.114.254:35573)
0925 22:59:09.440436 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.440441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.440801 (+ 360us) server_negotiation.cc:407] Connection header received
0925 22:59:09.441033 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.441036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.441099 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.441153 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.442032 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.442830 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.443964 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.444354 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.444459 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:09.447944 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:09.447948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:09.447952 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:09.447993 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:09.449965 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.450503 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.450507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.450509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.450600 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.450937 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.450941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.450943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.451197 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:09.451316 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:09.451693 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 22:59:09.451846 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:59:09.452284 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.435808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33852 (local address 127.20.114.252:40271)
0925 22:59:09.436433 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.436449 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.436489 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:59:09.437368 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.437389 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.437802 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.438145 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.438725 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.440352 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.441683 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.442392 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.442569 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:09.445342 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:09.445353 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:09.445380 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:09.445407 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:09.448236 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.448964 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.448969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.448974 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.449059 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.449426 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.449430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.449431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.449881 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:09.450929 (+ 1048us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:09.451000 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 22:59:09.451158 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":765,"server-negotiator.queue_time_us":489,"spinlock_wait_cycles":64256,"thread_start_us":169,"threads_started":1}
I20260925 22:59:09.452289 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.440239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.254:35573 (local address 127.0.0.1:39602)
0925 22:59:09.440671 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.440722 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:59:09.440915 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:59:09.441312 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:59:09.441317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:59:09.441724 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:59:09.441893 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:09.441900 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.442999 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.443003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:09.443827 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:09.443837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.444508 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.444512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:09.444620 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:09.445218 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:59:09.445265 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:59:09.447716 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:59:09.450080 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:09.450088 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:09.450094 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:09.450383 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:09.450723 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:09.450726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:09.450728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:09.450840 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:09.451328 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:59:09.451334 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:59:09.451449 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 22:59:09.451606 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 22:59:09.451791 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":3,"thread_start_us":131,"threads_started":1}
I20260925 22:59:09.452416 11349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.435808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33866 (local address 127.20.114.252:40271)
0925 22:59:09.436435 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.436450 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.436490 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:59:09.437383 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.437393 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.437805 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.438146 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.439213 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.440352 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.441729 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.442396 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.442951 (+ 555us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:09.446166 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:09.446170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:09.446173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:09.446205 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:09.448105 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.448715 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.448722 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.448730 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.448945 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.449355 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.449357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.449359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.449874 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:09.450104 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:09.450475 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 22:59:09.451251 (+ 776us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":189,"server-negotiator.queue_time_us":462,"spinlock_wait_cycles":17536,"thread_start_us":153,"threads_started":1}
I20260925 22:59:09.453675 11345 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:59:09.457553 11270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50a842aa7ad84143951fc569a5823078: Bootstrap starting.
Sep 25 22:59:09 dist-test-slave-x1lh krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377149, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.252@KRBTEST.COM for kudu/127.20.114.253@KRBTEST.COM
I20260925 22:59:09.461071 11270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50a842aa7ad84143951fc569a5823078: Neither blocks nor log segments found. Creating new log.
Sep 25 22:59:09 dist-test-slave-x1lh krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.254@KRBTEST.COM
I20260925 22:59:09.462137 11270 log.cc:824] T 00000000000000000000000000000000 P 50a842aa7ad84143951fc569a5823078: Log is configured to *not* fsync() on all Append() calls
I20260925 22:59:09.465039 11270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50a842aa7ad84143951fc569a5823078: No bootstrap required, opened a new log
I20260925 22:59:09.465190 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.454209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.253:39815 (local address 127.0.0.1:46430)
0925 22:59:09.454325 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.454340 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:59:09.454417 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:59:09.454873 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:59:09.455233 (+ 360us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:59:09.455460 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:59:09.455630 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:09.455638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.456826 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.456830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:09.457752 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:09.457764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.458369 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.458373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:09.458474 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:09.459022 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:59:09.459037 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:59:09.460793 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:59:09.463256 (+ 2463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:09.463260 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:09.463262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:09.463549 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:09.463945 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:09.463948 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:09.463950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:09.464040 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:09.464563 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:59:09.464566 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:59:09.464688 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 22:59:09.464802 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 22:59:09.465002 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 22:59:09.465692 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.454502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46430 (local address 127.20.114.253:39815)
0925 22:59:09.454642 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.454647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.454661 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:59:09.454709 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.454712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.454752 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.454857 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.455752 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.456595 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.457884 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.458251 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.458359 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:09.461059 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:09.461063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:09.461067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:09.461104 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:09.463118 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.463667 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.463670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.463672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.463755 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.464151 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.464157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.464159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.464437 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:09.464573 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:09.464920 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 22:59:09.465064 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:59:09.466434 11345 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:59:09.466964 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.455173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39608 (local address 127.20.114.254:35573)
0925 22:59:09.455289 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.455294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.455308 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:59:09.455438 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.455442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.455499 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.455578 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.456630 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.457231 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.458255 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.458625 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.458749 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:09.462138 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:09.462143 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:09.462147 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:09.462184 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:09.464391 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.465036 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.465041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.465044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.465153 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.465734 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.465739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.465743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.466011 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:09.466295 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:09.466654 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 22:59:09.466799 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:59:09.469455 11270 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50a842aa7ad84143951fc569a5823078 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5229d4e7383442cd8ad8740d802c5014" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35573 } } peers { permanent_uuid: "50a842aa7ad84143951fc569a5823078" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } } peers { permanent_uuid: "4bfa3908a18640e0bc470ed87083afea" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } }
I20260925 22:59:09.469744 11270 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50a842aa7ad84143951fc569a5823078 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50a842aa7ad84143951fc569a5823078, State: Initialized, Role: FOLLOWER
I20260925 22:59:09.470113 11270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50a842aa7ad84143951fc569a5823078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5229d4e7383442cd8ad8740d802c5014" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35573 } } peers { permanent_uuid: "50a842aa7ad84143951fc569a5823078" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } } peers { permanent_uuid: "4bfa3908a18640e0bc470ed87083afea" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } }
I20260925 22:59:09.471168 11352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50a842aa7ad84143951fc569a5823078 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5229d4e7383442cd8ad8740d802c5014" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35573 } } peers { permanent_uuid: "50a842aa7ad84143951fc569a5823078" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } } peers { permanent_uuid: "4bfa3908a18640e0bc470ed87083afea" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } } }
I20260925 22:59:09.471330 11352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50a842aa7ad84143951fc569a5823078 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:59:09.472077 11270 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50a842aa7ad84143951fc569a5823078 [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 22:59:09 dist-test-slave-x1lh krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377149, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.252@KRBTEST.COM for kudu/127.20.114.252@KRBTEST.COM
Sep 25 22:59:09 dist-test-slave-x1lh krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.114.253@KRBTEST.COM
I20260925 22:59:09.477010 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.466937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:40271 (local address 127.0.0.1:33880)
0925 22:59:09.467305 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.467322 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:59:09.467401 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:59:09.467892 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:59:09.467896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:59:09.468091 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:59:09.468164 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:09.468169 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.469394 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.469398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:09.470195 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:09.470206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.470804 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.470807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:09.470900 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:09.471465 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:59:09.471480 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:59:09.473063 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:59:09.475376 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:09.475379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:09.475382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:09.475654 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:09.475941 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:09.475944 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:09.475946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:09.476040 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:09.476483 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:59:09.476486 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:59:09.476567 (+ 81us) client_negotiation.cc:769] Sending connection context
0925 22:59:09.476655 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 22:59:09.476798 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":1}
I20260925 22:59:09.477890 11349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.467088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33880 (local address 127.20.114.252:40271)
0925 22:59:09.467663 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.467668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.467682 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:59:09.467719 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.467722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.467773 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.467878 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.468284 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.469258 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.470326 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.470676 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.470799 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:09.473224 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:09.473231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:09.473237 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:09.473269 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:09.475190 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.475752 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.475756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.475757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.475840 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.476136 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.476138 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.476140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.476360 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:09.476452 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:09.477588 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0925 22:59:09.477739 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":488}
I20260925 22:59:09.479530 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.468791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46436 (local address 127.20.114.253:39815)
0925 22:59:09.468970 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.468974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.468988 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:59:09.469026 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.469028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.469072 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.469179 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.469974 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.470633 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.471860 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.472140 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.472256 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:09.475409 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:09.475413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:09.475417 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:09.475445 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:09.477162 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.477697 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.477701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.477703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.477787 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.478102 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.478105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.478107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.478321 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:09.478490 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:09.479244 (+ 754us) server_negotiation.cc:299] Negotiation successful
0925 22:59:09.479378 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
W20260925 22:59:09.479951 11363 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50a842aa7ad84143951fc569a5823078: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:59:09.480031 11363 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:59:09.481202 11345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bfa3908a18640e0bc470ed87083afea: Bootstrap starting.
I20260925 22:59:09.477039 11201 master_runner.cc:186] Error getting master registration for 127.20.114.254:35573: OK, instance_id { permanent_uuid: "5229d4e7383442cd8ad8740d802c5014" instance_seqno: 1790377148933561 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:59:09.481668 11201 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:59:09.483379 20939 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:59:09.484375 11345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bfa3908a18640e0bc470ed87083afea: Neither blocks nor log segments found. Creating new log.
I20260925 22:59:09.485556 11345 log.cc:824] T 00000000000000000000000000000000 P 4bfa3908a18640e0bc470ed87083afea: Log is configured to *not* fsync() on all Append() calls
I20260925 22:59:09.485061 11168 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5229d4e7383442cd8ad8740d802c5014"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39608
I20260925 22:59:09.487977 11345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bfa3908a18640e0bc470ed87083afea: No bootstrap required, opened a new log
I20260925 22:59:09.491559 11345 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bfa3908a18640e0bc470ed87083afea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5229d4e7383442cd8ad8740d802c5014" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35573 } } peers { permanent_uuid: "50a842aa7ad84143951fc569a5823078" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } } peers { permanent_uuid: "4bfa3908a18640e0bc470ed87083afea" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } }
I20260925 22:59:09.491885 11345 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bfa3908a18640e0bc470ed87083afea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bfa3908a18640e0bc470ed87083afea, State: Initialized, Role: FOLLOWER
I20260925 22:59:09.492300 11345 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bfa3908a18640e0bc470ed87083afea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5229d4e7383442cd8ad8740d802c5014" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35573 } } peers { permanent_uuid: "50a842aa7ad84143951fc569a5823078" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } } peers { permanent_uuid: "4bfa3908a18640e0bc470ed87083afea" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } }
I20260925 22:59:09.493186 11366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bfa3908a18640e0bc470ed87083afea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5229d4e7383442cd8ad8740d802c5014" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35573 } } peers { permanent_uuid: "50a842aa7ad84143951fc569a5823078" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } } peers { permanent_uuid: "4bfa3908a18640e0bc470ed87083afea" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } } }
I20260925 22:59:09.493506 11366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bfa3908a18640e0bc470ed87083afea [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:59:09.493894 11345 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bfa3908a18640e0bc470ed87083afea [sys.catalog]: configured and running, proceeding with master startup.
W20260925 22:59:09.500707 11377 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4bfa3908a18640e0bc470ed87083afea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:59:09.500813 11377 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:59:09.498929 11277 master_runner.cc:186] Error getting master registration for 127.20.114.254:35573: OK, instance_id { permanent_uuid: "5229d4e7383442cd8ad8740d802c5014" instance_seqno: 1790377148933561 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:59:09.503065 11277 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:59:09.580243 11194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } attempt: 4
Sep 25 22:59:09 dist-test-slave-x1lh krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790377148, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.114.254@KRBTEST.COM for kudu/127.20.114.252@KRBTEST.COM
I20260925 22:59:09.592489 11274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.580953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.114.252:40271 (local address 127.0.0.1:33884)
0925 22:59:09.581149 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 22:59:09.581168 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:59:09.581297 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:59:09.581859 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:59:09.581863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:59:09.582145 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:59:09.582330 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:09.582339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.583442 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.583449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:09.584469 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:59:09.584484 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.585145 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.585150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:59:09.585299 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:09.586001 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:59:09.586019 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:59:09.588002 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:59:09.590485 (+ 2483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:09.590490 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:09.590493 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:09.590799 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:09.591153 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:59:09.591156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:59:09.591158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:59:09.591263 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:59:09.591808 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:59:09.591811 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:59:09.591935 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 22:59:09.592053 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 22:59:09.592257 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 22:59:09.593259 11349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.581485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33884 (local address 127.20.114.252:40271)
0925 22:59:09.581615 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.581620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.581635 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:59:09.581681 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.581684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.581730 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.581817 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.582501 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.583305 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.584636 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.585017 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.585133 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:09.588205 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:09.588210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:09.588214 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:09.588248 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:09.590332 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.590931 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.590937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.590939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.591049 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.591365 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.591368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.591371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.591634 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:09.591744 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:09.592900 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0925 22:59:09.593053 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:59:09.596972 11194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014: Bootstrap starting.
I20260925 22:59:09.600401 11194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014: Neither blocks nor log segments found. Creating new log.
I20260925 22:59:09.601536 11194 log.cc:824] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014: Log is configured to *not* fsync() on all Append() calls
I20260925 22:59:09.604636 11194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014: No bootstrap required, opened a new log
I20260925 22:59:09.609102 11194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5229d4e7383442cd8ad8740d802c5014" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35573 } } peers { permanent_uuid: "50a842aa7ad84143951fc569a5823078" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } } peers { permanent_uuid: "4bfa3908a18640e0bc470ed87083afea" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } }
I20260925 22:59:09.609457 11194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5229d4e7383442cd8ad8740d802c5014, State: Initialized, Role: FOLLOWER
I20260925 22:59:09.609957 11194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5229d4e7383442cd8ad8740d802c5014" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35573 } } peers { permanent_uuid: "50a842aa7ad84143951fc569a5823078" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } } peers { permanent_uuid: "4bfa3908a18640e0bc470ed87083afea" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } }
I20260925 22:59:09.611626 11194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:59:09.611752 11168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 22:59:09.611935 11168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:59:09.613586 11168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5229d4e7383442cd8ad8740d802c5014" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35573 } } peers { permanent_uuid: "50a842aa7ad84143951fc569a5823078" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } } peers { permanent_uuid: "4bfa3908a18640e0bc470ed87083afea" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } }
I20260925 22:59:09.616341 11243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5229d4e7383442cd8ad8740d802c5014" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "50a842aa7ad84143951fc569a5823078"
I20260925 22:59:09.616665 11243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50a842aa7ad84143951fc569a5823078 [term 0 FOLLOWER]: Advancing to term 1
W20260925 22:59:09.618489 11391 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:59:09.618605 11391 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:59:09.618944 11380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5229d4e7383442cd8ad8740d802c5014" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35573 } } peers { permanent_uuid: "50a842aa7ad84143951fc569a5823078" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } } peers { permanent_uuid: "4bfa3908a18640e0bc470ed87083afea" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } } }
I20260925 22:59:09.619091 11380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:59:09.619799 11243 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50a842aa7ad84143951fc569a5823078 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5229d4e7383442cd8ad8740d802c5014 in term 1.
I20260925 22:59:09.621826 11143 leader_election.cc:302] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50a842aa7ad84143951fc569a5823078, 5229d4e7383442cd8ad8740d802c5014; no voters:
I20260925 22:59:09.626142 11319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5229d4e7383442cd8ad8740d802c5014" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4bfa3908a18640e0bc470ed87083afea"
I20260925 22:59:09.626466 11319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bfa3908a18640e0bc470ed87083afea [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:59:09.628010 11319 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4bfa3908a18640e0bc470ed87083afea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5229d4e7383442cd8ad8740d802c5014 in term 1.
I20260925 22:59:09.628504 11168 leader_election.cc:288] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014 [CANDIDATE]: Term 1 election: Requested vote from peers 50a842aa7ad84143951fc569a5823078 (127.20.114.253:39815), 4bfa3908a18640e0bc470ed87083afea (127.20.114.252:40271)
I20260925 22:59:09.630956 11380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:59:09.631283 11126 master_runner.cc:263] No leader master found from master 5229d4e7383442cd8ad8740d802c5014
I20260925 22:59:09.631345 11380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014 [term 1 LEADER]: Becoming Leader. State: Replica: 5229d4e7383442cd8ad8740d802c5014, State: Running, Role: LEADER
I20260925 22:59:09.631385 11126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:59:09.631757 11380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5229d4e7383442cd8ad8740d802c5014" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35573 } } peers { permanent_uuid: "50a842aa7ad84143951fc569a5823078" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } } peers { permanent_uuid: "4bfa3908a18640e0bc470ed87083afea" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } }
I20260925 22:59:09.634100 11380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5229d4e7383442cd8ad8740d802c5014. Latest consensus state: current_term: 1 leader_uuid: "5229d4e7383442cd8ad8740d802c5014" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5229d4e7383442cd8ad8740d802c5014" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35573 } } peers { permanent_uuid: "50a842aa7ad84143951fc569a5823078" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } } peers { permanent_uuid: "4bfa3908a18640e0bc470ed87083afea" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } } }
I20260925 22:59:09.634385 11380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014 [sys.catalog]: This master's current role is: LEADER
I20260925 22:59:09.642949 11399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:59:09.644186 11399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:59:09.647676 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.637433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39616 (local address 127.20.114.254:35573)
0925 22:59:09.637560 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.637565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.637874 (+ 309us) server_negotiation.cc:407] Connection header received
0925 22:59:09.638094 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.638098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.638148 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.638204 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.639288 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.640063 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.641327 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.641623 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.641698 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:09.643758 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:09.643762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:09.643765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:09.643793 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:09.645694 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.646233 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.646236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.646238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.646326 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.646664 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.646668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.646669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.646897 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:09.647005 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:09.647389 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 22:59:09.647517 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:59:09.658365 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.641317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46448 (local address 127.20.114.253:39815)
0925 22:59:09.641470 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.641475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.648568 (+ 7093us) server_negotiation.cc:407] Connection header received
0925 22:59:09.648773 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.648777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.648834 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.648942 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.649940 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.650684 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.651804 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.652165 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.652255 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:09.654275 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:09.654279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:09.654282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:09.654318 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:09.656248 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.656828 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.656832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.656834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.656971 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.657308 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.657311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.657312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.657563 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:09.657672 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:09.658058 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 22:59:09.658192 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 22:59:09.661753 11318 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4bfa3908a18640e0bc470ed87083afea [term 1 FOLLOWER]: Refusing update from remote peer 5229d4e7383442cd8ad8740d802c5014: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:59:09.662762 11397 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bfa3908a18640e0bc470ed87083afea" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:59:09.663687 11349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:09.637996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33898 (local address 127.20.114.252:40271)
0925 22:59:09.638126 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.638131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.642261 (+ 4130us) server_negotiation.cc:407] Connection header received
0925 22:59:09.642493 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.642496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.642555 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.642614 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.644459 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.645353 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.657001 (+ 11648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:09.657466 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:09.657585 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:09.659623 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:59:09.659628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:59:09.659630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:59:09.659662 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:59:09.661489 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.662130 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.662133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.662135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.662232 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:59:09.662540 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:59:09.662543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:59:09.662545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:59:09.662785 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:59:09.662940 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:09.663399 (+ 459us) server_negotiation.cc:299] Negotiation successful
0925 22:59:09.663530 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:59:09.668413 11366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bfa3908a18640e0bc470ed87083afea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5229d4e7383442cd8ad8740d802c5014. Latest consensus state: current_term: 1 leader_uuid: "5229d4e7383442cd8ad8740d802c5014" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5229d4e7383442cd8ad8740d802c5014" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35573 } } peers { permanent_uuid: "50a842aa7ad84143951fc569a5823078" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } } peers { permanent_uuid: "4bfa3908a18640e0bc470ed87083afea" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } } }
I20260925 22:59:09.668565 11366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bfa3908a18640e0bc470ed87083afea [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:59:09.670126 11243 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50a842aa7ad84143951fc569a5823078 [term 1 FOLLOWER]: Refusing update from remote peer 5229d4e7383442cd8ad8740d802c5014: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:59:09.670948 11397 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50a842aa7ad84143951fc569a5823078" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 22:59:09.672408 11298 tablet.cc:2369] T 00000000000000000000000000000000 P 4bfa3908a18640e0bc470ed87083afea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:59:09.675292 11222 tablet.cc:2369] T 00000000000000000000000000000000 P 50a842aa7ad84143951fc569a5823078: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:59:09.675642 11352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50a842aa7ad84143951fc569a5823078 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5229d4e7383442cd8ad8740d802c5014. Latest consensus state: current_term: 1 leader_uuid: "5229d4e7383442cd8ad8740d802c5014" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5229d4e7383442cd8ad8740d802c5014" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35573 } } peers { permanent_uuid: "50a842aa7ad84143951fc569a5823078" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } } peers { permanent_uuid: "4bfa3908a18640e0bc470ed87083afea" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } } }
I20260925 22:59:09.675755 11352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50a842aa7ad84143951fc569a5823078 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:59:09.678467 11397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5229d4e7383442cd8ad8740d802c5014" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5229d4e7383442cd8ad8740d802c5014" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35573 } } peers { permanent_uuid: "50a842aa7ad84143951fc569a5823078" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } } peers { permanent_uuid: "4bfa3908a18640e0bc470ed87083afea" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } } }
I20260925 22:59:09.678653 11397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014 [sys.catalog]: This master's current role is: LEADER
I20260925 22:59:09.680011 11366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bfa3908a18640e0bc470ed87083afea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5229d4e7383442cd8ad8740d802c5014" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5229d4e7383442cd8ad8740d802c5014" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35573 } } peers { permanent_uuid: "50a842aa7ad84143951fc569a5823078" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } } peers { permanent_uuid: "4bfa3908a18640e0bc470ed87083afea" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } } }
I20260925 22:59:09.680162 11366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bfa3908a18640e0bc470ed87083afea [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:59:09.680423 11404 mvcc.cc:205] Tried to move back new op lower bound from 7333384805003526144 to 7333384804896452608. Current Snapshot: MvccSnapshot[applied={T|T < 7333384805003526144}]
I20260925 22:59:09.683487 11380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5229d4e7383442cd8ad8740d802c5014" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5229d4e7383442cd8ad8740d802c5014" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35573 } } peers { permanent_uuid: "50a842aa7ad84143951fc569a5823078" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } } peers { permanent_uuid: "4bfa3908a18640e0bc470ed87083afea" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } } }
I20260925 22:59:09.683641 11380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014 [sys.catalog]: This master's current role is: LEADER
I20260925 22:59:09.685122 11352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50a842aa7ad84143951fc569a5823078 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5229d4e7383442cd8ad8740d802c5014" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5229d4e7383442cd8ad8740d802c5014" member_type: VOTER last_known_addr { host: "127.20.114.254" port: 35573 } } peers { permanent_uuid: "50a842aa7ad84143951fc569a5823078" member_type: VOTER last_known_addr { host: "127.20.114.253" port: 39815 } } peers { permanent_uuid: "4bfa3908a18640e0bc470ed87083afea" member_type: VOTER last_known_addr { host: "127.20.114.252" port: 40271 } } }
I20260925 22:59:09.685261 11352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50a842aa7ad84143951fc569a5823078 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:59:09.685812 11399 catalog_manager.cc:1347] Generated new cluster ID: a8187c33f92b4eec84a3f5e0dba67230
I20260925 22:59:09.685882 11399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:59:09.686060 11406 mvcc.cc:205] Tried to move back new op lower bound from 7333384805003526144 to 7333384804896452608. Current Snapshot: MvccSnapshot[applied={T|T < 7333384805003526144}]
I20260925 22:59:09.716572 11399 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:59:09.718086 11399 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:59:09.728497 11399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5229d4e7383442cd8ad8740d802c5014: Generated new TSK 0
I20260925 22:59:09.729588 11399 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:59:09.762903 11269 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:09.761150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46464 (local address 127.20.114.253:39815)
0925 22:59:09.761355 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.761362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.761987 (+ 625us) server_negotiation.cc:407] Connection header received
0925 22:59:09.762037 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.762039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.762092 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.762164 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.762758 (+ 594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46464: BlockingRecv error: recv got EOF from 127.0.0.1:46464 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
W20260925 22:59:09.762903 11198 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:09.761083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39628 (local address 127.20.114.254:35573)
0925 22:59:09.761228 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.761234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.761618 (+ 384us) server_negotiation.cc:407] Connection header received
0925 22:59:09.761874 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.761878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.761938 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.761996 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.762759 (+ 763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39628: BlockingRecv error: recv got EOF from 127.0.0.1:39628 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 22:59:09.763237 11349 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:09.761737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33912 (local address 127.20.114.252:40271)
0925 22:59:09.761856 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.761862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.762090 (+ 228us) server_negotiation.cc:407] Connection header received
0925 22:59:09.762404 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.762408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.762464 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.762520 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.763108 (+ 588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33912: BlockingRecv error: recv got EOF from 127.0.0.1:33912 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 22:59:09.767340 11269 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:09.765689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46480 (local address 127.20.114.253:39815)
0925 22:59:09.765825 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.765829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.766260 (+ 431us) server_negotiation.cc:407] Connection header received
0925 22:59:09.766400 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.766403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.766480 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.766557 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.767184 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46480: BlockingRecv error: recv got EOF from 127.0.0.1:46480 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 22:59:09.772023 11269 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:09.771092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46496 (local address 127.20.114.253:39815)
0925 22:59:09.771219 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.771222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.771235 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:59:09.771291 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.771294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.771341 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.771436 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.771890 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46496: BlockingRecv error: recv got EOF from 127.0.0.1:46496 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 22:59:09.777797 11269 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:09.776366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46500 (local address 127.20.114.253:39815)
0925 22:59:09.776524 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.776528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.776667 (+ 139us) server_negotiation.cc:407] Connection header received
0925 22:59:09.776813 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.776815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.776981 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.777062 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.777627 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46500: BlockingRecv error: recv got EOF from 127.0.0.1:46500 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 22:59:09.785887 11269 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:09.784433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46512 (local address 127.20.114.253:39815)
0925 22:59:09.784635 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.784639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.784777 (+ 138us) server_negotiation.cc:407] Connection header received
0925 22:59:09.784974 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.784977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.785032 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.785110 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.785696 (+ 586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46512: BlockingRecv error: recv got EOF from 127.0.0.1:46512 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
W20260925 22:59:09.798254 11269 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:09.796584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46526 (local address 127.20.114.253:39815)
0925 22:59:09.796749 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.796754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.797032 (+ 278us) server_negotiation.cc:407] Connection header received
0925 22:59:09.797161 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.797164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.797231 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.797513 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.798114 (+ 601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46526: BlockingRecv error: recv got EOF from 127.0.0.1:46526 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 22:59:09.817975 11269 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:09.816664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46532 (local address 127.20.114.253:39815)
0925 22:59:09.816820 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.816826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.817026 (+ 200us) server_negotiation.cc:407] Connection header received
0925 22:59:09.817156 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.817160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.817223 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.817333 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.817839 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46532: BlockingRecv error: recv got EOF from 127.0.0.1:46532 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 22:59:09.853828 11269 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:09.852450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46548 (local address 127.20.114.253:39815)
0925 22:59:09.852635 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.852639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.852655 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:59:09.852804 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.852807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.852907 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.853011 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.853666 (+ 655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46548: BlockingRecv error: recv got EOF from 127.0.0.1:46548 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260925 22:59:09.921357 11269 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:09.920047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46550 (local address 127.20.114.253:39815)
0925 22:59:09.920230 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:09.920235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:09.920344 (+ 109us) server_negotiation.cc:407] Connection header received
0925 22:59:09.920435 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:09.920438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:09.920491 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:09.920590 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:09.921249 (+ 659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46550: BlockingRecv error: recv got EOF from 127.0.0.1:46550 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260925 22:59:10.053159 11269 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:10.051838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46560 (local address 127.20.114.253:39815)
0925 22:59:10.052012 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:10.052016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:10.052193 (+ 177us) server_negotiation.cc:407] Connection header received
0925 22:59:10.052318 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:10.052321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:10.052376 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:10.052464 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:10.053029 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46560: BlockingRecv error: recv got EOF from 127.0.0.1:46560 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260925 22:59:10.313060 11269 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:59:10.311537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46574 (local address 127.20.114.253:39815)
0925 22:59:10.311683 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:10.311686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:10.311755 (+ 69us) server_negotiation.cc:407] Connection header received
0925 22:59:10.311903 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:10.311907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:10.311965 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:10.312051 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:59:10.312921 (+ 870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46574: BlockingRecv error: recv got EOF from 127.0.0.1:46574 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:59:10.481576 11363 catalog_manager.cc:1259] Loaded cluster ID: a8187c33f92b4eec84a3f5e0dba67230
I20260925 22:59:10.481673 11363 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 50a842aa7ad84143951fc569a5823078: loading cluster ID for follower catalog manager: success
I20260925 22:59:10.483590 11363 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 50a842aa7ad84143951fc569a5823078: acquiring CA information for follower catalog manager: success
I20260925 22:59:10.484998 11363 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 50a842aa7ad84143951fc569a5823078: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:59:10.502445 11377 catalog_manager.cc:1259] Loaded cluster ID: a8187c33f92b4eec84a3f5e0dba67230
I20260925 22:59:10.502519 11377 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4bfa3908a18640e0bc470ed87083afea: loading cluster ID for follower catalog manager: success
I20260925 22:59:10.504508 11377 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4bfa3908a18640e0bc470ed87083afea: acquiring CA information for follower catalog manager: success
I20260925 22:59:10.505893 11377 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4bfa3908a18640e0bc470ed87083afea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:59:10.831792 11476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:10.827416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46588 (local address 127.20.114.253:39815)
0925 22:59:10.827710 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:10.827715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:10.827733 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:59:10.827795 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:10.827798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:10.827977 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:10.828144 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:59:10.828460 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:10.829156 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:10.830102 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:10.830352 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:10.830451 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:10.830823 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:59:10.831321 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:59:10.831422 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:10.831578 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 22:59:10.831646 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":120,"threads_started":1}
I20260925 22:59:10.840952 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:10.836048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33922 (local address 127.20.114.252:40271)
0925 22:59:10.836368 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:10.836373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:10.836390 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:59:10.836617 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:10.836620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:10.836763 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:10.836959 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:59:10.837373 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:10.837960 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:10.838973 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:10.839215 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:10.839305 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:10.839771 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:59:10.840315 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:59:10.840450 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:10.840595 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 22:59:10.840664 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":110,"threads_started":1}
I20260925 22:59:10.849324 11476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:10.844364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46602 (local address 127.20.114.253:39815)
0925 22:59:10.844502 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:10.844507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:10.845281 (+ 774us) server_negotiation.cc:407] Connection header received
0925 22:59:10.845356 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:10.845362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:10.845444 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:10.845538 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:59:10.845971 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:10.846586 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:10.847819 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:10.848224 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:10.848339 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:10.848666 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:59:10.848781 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:59:10.848830 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:10.849039 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 22:59:10.849120 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:59:10.849534 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:10.844920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33928 (local address 127.20.114.252:40271)
0925 22:59:10.845032 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:10.845036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:10.845466 (+ 430us) server_negotiation.cc:407] Connection header received
0925 22:59:10.845669 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:10.845676 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:10.845743 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:10.845808 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:59:10.846040 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:10.846909 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:10.848112 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:10.848477 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:10.848611 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:10.848961 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:59:10.849079 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:59:10.849156 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:10.849329 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 22:59:10.849401 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 22:59:10.850350 11488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:59:10.844427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39642 (local address 127.20.114.254:35573)
0925 22:59:10.844725 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0925 22:59:10.844730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:59:10.844896 (+ 166us) server_negotiation.cc:407] Connection header received
0925 22:59:10.845138 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:59:10.845142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:59:10.845342 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:59:10.845494 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:59:10.845740 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:10.846777 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:10.847942 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:59:10.848258 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:59:10.848386 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:59:10.848823 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:59:10.849292 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:59:10.850081 (+ 789us) server_negotiation.cc:1035] Waiting for connection context
0925 22:59:10.850124 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 22:59:10.850209 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":87,"threads_started":1}
I20260925 22:59:10.853971 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 11126
I20260925 22:59:10.861024 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 11201
I20260925 22:59:10.868273 20939 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV8JYeA/build/debug/bin/kudu with pid 11277
2026-09-25T22:59:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3365 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6633 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (294771 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST